Lead Data Engineer - Data Ingestion
About The Position
The Lead Data Engineer serves as both a technical authority and individual contributor responsible for the delivery and continuous improvement of healthcare data products and platform components. This position requires a proven track record of architecting secure, high-throughput data pipelines and leading high-performing teams of 5-8 engineers. The Lead Data Engineer translates business strategy into data engineering execution, guiding the team in the adoption of tools such as AWS Glue, Iceberg, SageMaker, and feature flag frameworks, while ensuring HIPAA and HITRUST compliance. Acting as a bridge between engineering, data science, and product functions, this role supports alignment during sprint planning and delivery, emphasizing clear communication, documentation, and team empowerment.
Requirements
- Excellent written, verbal, and interpersonal communication skills.
- Strong organizational skills and attention to detail.
- Excellent documentation, mentoring, and leadership abilities.
- Proven ability to motivate and lead teams, setting clear expectations and driving accountability.
- Ability to manage multiple concurrent projects and priorities in a dynamic environment.
- Skilled at communicating complex technical topics clearly and effectively to various audiences.
- Strong problem-solving and analytical thinking skills.
- Ability to operate effectively in collaborative, cross-functional environments.
- Programming Languages: Python, PySpark, Scala (10+ years) - Directed large-scale data platform development with an emphasis on modularity and performance.
- Orchestration Tools: Airflow (6+ years) - Defined standards for DAG design, observability, and reusability across teams.
- Cloud Platforms: AWS (Glue, EMR, Lambda, Step Functions) (5+ years) - Designed hybrid orchestration strategies using serverless and EMR-based pipelines.
- Data Architecture: Iceberg (4-5 years) - Led lakehouse strategy including governance, versioning, and cost optimization.
- Architecture Leadership: Deep expertise in designing and optimizing data lake and lakehouse architectures at scale, including governance, schema evolution, cost efficiency, and access control.
- Containerization: Docker/Kubernetes (6-8 years) - Managed Kubernetes-based orchestration for enterprise data pipelines and established Docker standards aligned with compliance and DevOps strategies.
- Team Leadership: Directed a data engineering team of 5-8 engineers, driving architectural decisions, enforcing engineering standards, and championing Agile delivery practices within a high-performance environment.
- Bachelor's degree in computer science, Engineering, or a related field, or equivalent combination of education and experience.
- 10+ years of experience in data engineering roles.
- Extensive experience with large-scale data systems and cloud-based architectures (AWS required).
Nice To Haves
- Prior experience with healthcare or regulated datasets strongly preferred.
Responsibilities
- Architect and lead the adoption of serverless data platforms integrating AWS services such as Glue, Lambda, DynamoDB, and Lakehouse Formation.
- Lead GitOps strategy for infrastructure and pipeline deployment, ensuring compliance, rollback safety, and peer-reviewed workflows.
- Establish and enforce best practices for feature flag toggling strategies in critical data pipelines.
- Drive internal knowledge sharing and technical documentation standards across the engineering organization.
- Mentor and guide junior engineers to promote professional growth, collaboration, and best practices.
- Design scalable architectural solutions and effectively communicate complex technical concepts to both technical and non-technical audiences.
- Own technical environment and tooling strategy across the data engineering organization, ensuring standards, documentation, and collaboration practices are consistently applied.
- Maintain and oversee governance using: GitHub (repository standards and peer review) AWS Console (Glue, Iceberg, EMR, IAM governance) Jupyter/Python IDE (advanced notebook-based development) Lucidchart (platform-wide architecture diagrams) Microsoft Teams/Zoom (engineering leadership and stakeholder communications) Jira & Confluence (governance documentation and delivery tracking)
Benefits
- health and welfare benefits
- 401(K) savings with employer matching
- paid time off
- life, AD&D and disability insurances
- various ancillary benefits, such as pet insurance, virtual physical therapy, and employee assistance programs
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Industry
Professional, Scientific, and Technical Services
Number of Employees
1,001-5,000 employees
