Lead AWS Data Engineer

Tech Mahindra•Saint Louis, MO

3d•$120,000 - $135,000

About The Position

Position Title: Lead AWS Data Engineer Location: St. Louis, MO • Design and build scalable, cloud-native data platforms using PySpark, Python, and modern data engineering practices. • Mentor and guide other engineers, sharing knowledge, reviewing code, and fostering a culture of curiosity, growth, and continuous improvement. • Create robust, maintainable ETL/ELT pipelines that integrate with diverse systems and serve business-critical use cases. • Lead by example—write high-quality, testable code and participate in architecture and design discussions with a long-term view in mind. • Decompose complex problems into modular, efficient, and scalable components that align with platform and product goals. • Champion best practices in data engineering, including testing, version control, documentation, and performance tuning. • Drive collaboration across teams, working closely with product managers, data scientists, and other engineers to deliver high-impact solutions. • Support data governance and quality efforts, ensuring data lineage, cataloging, and access management are built into the platform. • Continuously learn and apply new technologies, frameworks, and tools to improve team productivity and platform reliability. • Own and optimize cloud infrastructure components related to data engineering workflows, storage, processing, and orchestration. • Participate in architectural discussions, iteration planning, and feature sizing meetings • Adhere to Agile processes and participate actively in agile ceremonies • Stakeholder management skills

Requirements

Solid understanding of EMR Serverless, Apache Airflow, DAGs, Iceberg S3, SQS, DLQ, Lambda, AWS Lake Formation, Glue, Athena, Data Pipelines, Databricks Unity catalog, DBX S3
Any experience in YAML Data Contracts
Experience integrating heterogeneous systems and building resilient data pipelines across cloud environments.

Nice To Haves

Experience with containerization and infrastructure automation (e.g., Docker, Terraform) will be a good addition.
Familiarity with orchestration tools (e.g., Airflow, dbt, Step Functions, etc.).
Exposure to data governance tools and practices (e.g., Lake Formation, Purview, or Atlan).
AWS Certified Data Analytics, Azure Data Engineer), or demonstrable contributions to open source/data engineering communities will be a bonus.
Exposure to machine learning data pipelines or MLOps is a plus.

Responsibilities

Design and build scalable, cloud-native data platforms using PySpark, Python, and modern data engineering practices.
Mentor and guide other engineers, sharing knowledge, reviewing code, and fostering a culture of curiosity, growth, and continuous improvement.
Create robust, maintainable ETL/ELT pipelines that integrate with diverse systems and serve business-critical use cases.
Lead by example—write high-quality, testable code and participate in architecture and design discussions with a long-term view in mind.
Decompose complex problems into modular, efficient, and scalable components that align with platform and product goals.
Champion best practices in data engineering, including testing, version control, documentation, and performance tuning.
Drive collaboration across teams, working closely with product managers, data scientists, and other engineers to deliver high-impact solutions.
Support data governance and quality efforts, ensuring data lineage, cataloging, and access management are built into the platform.
Continuously learn and apply new technologies, frameworks, and tools to improve team productivity and platform reliability.
Own and optimize cloud infrastructure components related to data engineering workflows, storage, processing, and orchestration.
Participate in architectural discussions, iteration planning, and feature sizing meetings
Adhere to Agile processes and participate actively in agile ceremonies
Stakeholder management skills