Lead - Data Engineer
About The Position
Astrana Health is seeking a Lead - Data Engineer to lead the hands-on design, development, and operation of key components of its enterprise data platform, including the Constellation Lakehouse, Connect API (GraphQL) layer, and Lakebase-backed applications. The role builds scalable data pipelines, integrates healthcare data sources, and develops unified data models supporting analytics, reporting, AI and machine learning, and operational applications. It combines advanced data engineering with responsibility for platform reliability, performance, security, and automation in an Azure Databricks environment. As a hands-on technical lead, this position translates complex requirements into reliable production solutions, establishes engineering standards, and mentors engineers. It works with analytics, software engineering, infrastructure, security, business stakeholders, and external vendors to advance the platform and support its adoption.
Requirements
- Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.
- At least 5 years of data engineering experience building and supporting enterprise data platforms and pipelines.
- At least 3 years of hands-on Databricks experience in production environments.
- Strong proficiency in Python, PySpark, SQL, and Apache Spark.
- Experience with Delta Lake and modern lakehouse architectures.
- Hands-on experience with Azure data services and cloud-based data platforms.
- Strong knowledge of data modeling, relational databases, and ETL/ELT best practices.
- Experience integrating enterprise databases and APIs into scalable data solutions.
- Experience with Git, CI/CD pipelines, and automated deployment practices.
- Ability to monitor, troubleshoot, and optimize data pipelines and platform performance.
- Demonstrated leadership in solution design, code reviews, mentoring, and cross-functional collaboration.
- Strong problem-solving and communication skills.
Nice To Haves
- Experience administering and optimizing Databricks environments, including Unity Catalog.
- Experience with PostgreSQL, GraphQL APIs, and operational applications leveraging lakehouse data.
- Healthcare data experience, including claims, eligibility, provider, or clinical datasets.
- Familiarity with healthcare interoperability standards such as HL7, FHIR, and X12.
- Experience with Terraform or other infrastructure-as-code tools.
- Experience with streaming data pipelines and event-driven architectures.
- Experience supporting AI, machine learning, or NLP data workloads.
- Familiarity with regulated data environments and governance best practices.
- Experience with cloud data platform migrations and modernization initiatives.
- Azure, Databricks, or AWS certifications.
Responsibilities
- Lead the design, development, and delivery of the Constellation Lakehouse in Databricks, the Connect API (GraphQL) layer, and Lakebase-backed applications.
- Build and maintain scalable batch and incremental data pipelines using Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
- Integrate transactional databases, APIs, files, and external partner data into governed datasets for analytics and operational applications.
- Design and maintain unified healthcare data models spanning membership, providers, claims, encounters, clinical data, and other business domains.
- Develop ingestion and transformation processes for change data capture, schema evolution, deduplication, historical tracking, and reconciliation.
- Build and optimize data access layers and APIs with software engineering teams to support consistent business definitions and reliable downstream consumption.
- Develop and support Lakebase-backed applications and PostgreSQL data structures, including integration with lakehouse datasets and application services.
- Maintain and optimize Databricks workspaces, compute resources, job orchestration, and environment configurations with infrastructure and security teams.
- Implement data governance and access controls through Unity Catalog and organizational standards, including permissions, lineage, and auditability.
- Establish automated data quality checks, source-to-target reconciliation, freshness monitoring, and alerting.
- Apply DevOps practices, including Git, automated testing, CI/CD, infrastructure as code, and repeatable deployment across development, testing, and production.
- Tune pipelines, Spark workloads, SQL queries, and compute configurations to improve performance, scalability, and cost efficiency.
- Investigate and resolve platform incidents and pipeline failures, perform root-cause analysis, and implement preventive improvements.
- Partner with infrastructure teams on backup, recovery, and business continuity procedures, and maintain operational documentation and runbooks.
- Conduct code and design reviews, establish engineering standards, and mentor engineers across internal and vendor teams.
Benefits
- The position is remotely based in the U.S.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
