AWS Data Engineer

Agama Solutions•San Francisco, CA

21h•Hybrid

About The Position

As an AWS Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure. You will be responsible for onboarding data sources into our data lake, building robust data pipelines, and ensuring data quality and governance. Your expertise in AWS services and data engineering best practices will be essential in driving our data initiatives forward.

Requirements

Proficiency in AWS Lake Formation, Step Functions, Lambda (serverless), EC2, EMR, and EKS.
Strong experience with Python and Terraform scripting.
Experience with Jupyter Notebook, RDS, Snowflake, and Iceberg table formats.
Expertise in Spark and data pipeline orchestration tools like Airflow and dbt.
Solid understanding of data engineering principles, including ETL processes, data warehousing, and data modeling.
Knowledge of data governance principles and best practices.
Strong analytical and problem-solving skills with the ability to troubleshoot and resolve data-related issues.
Excellent communication skills with the ability to collaborate effectively with cross-functional teams.
Only w2 Required

Nice To Haves

AWS Certified Data Engineer or Analytics – Specialty, AWS Certified Solutions Architect, or other relevant certifications.
Previous experience in a similar role within a fast-paced, data-driven environment.

Responsibilities

Onboard various data sources into the data lake, ensuring seamless integration and data consistency.
Design, develop, and maintain scalable and efficient data pipelines using AWS services such as Lambda, Step Functions, and EMR.
Register data sources and manage metadata to ensure data discoverability and accessibility.
Implement data quality checks and transformations to ensure the accuracy and reliability of data.
Comply with data governance principles and best practices to ensure data security, privacy, and compliance.
Utilize Terraform scripting to manage and automate AWS infrastructure.
Leverage Spark and other big data technologies to process and analyze large datasets.
Use Airflow and Step Functions to orchestrate complex data workflows.
Work with Snowflake, Iceberg table formats, and other data modeling tools to design and optimize data storage solutions.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.