Senior Data Engineer, Platform

Vanna Health, Inc

2d•$180,000 - $220,000

About The Position

Vanna Health is hiring a Senior Data Engineer to own the infrastructure and technical direction of its data platform. The role covers how data jobs run and are optimized; orchestration and deployment; connectivity to partners and systems; and platform reliability and observability. The engineer will make architecture decisions and build the platform as tested, version-controlled software using infrastructure as code. This is a senior individual contributor position with a path to leading data engineering as the team grows. The platform supports both health-plan data about members and data generated by Vanna Health’s own software. The role is complementary to analytics engineering: analytics engineering builds curated gold-tier tables, while this position ensures those tables and their source data arrive complete, on time, reliably, and at a sensible cost. Key areas of focus in the first year are compute, ingestion, and orchestration, with AI and governance work growing as the team develops. The position includes scaling batch processing from single-machine workloads to distributed compute, migrating legacy Databricks jobs, connecting payer data feeds end to end, and reviewing scheduled jobs for appropriate compute choices and cost. It also involves automating recurring monthly data exchanges with partners, making data usable by AI through governed definitions and tools, protecting PHI, and collaborating with coaches, clinicians, researchers, partners, and colleagues. Vanna Health is a small, AI-forward team that builds and evaluates AI systems using real data.

Requirements

Typically seven or more years building production software or data systems, including several years owning data infrastructure or pipelines end to end, from design through operation.
Hands-on production experience at scale with at least one distributed data processing engine, such as Spark, Databricks, Snowflake, BigQuery, Flink, Dask, or Ray.
Ability to distinguish workloads suited to a single machine from those requiring distributed processing, and experience making large jobs fast and memory-predictable with demonstrable before-and-after results.
Strong Python skills and defensible SQL skills, including the ability to explain and validate query results.
Experience designing systems on a major cloud, with AWS preferred, including storage, compute, networking, IAM, and cost.
Production engineering discipline, including automated testing, CI/CD, infrastructure as code, code review, observability, and incident response.
Experience leading technical work by setting project or team direction and mentoring other engineers.
Experience investigating data problems that originate upstream and tracing them to their source.
Daily use of AI coding agents in engineering work, with the ability to describe an incorrect AI result encountered on real data and how it was detected.

Nice To Haves

Experience with Databricks, Delta Lake, Unity Catalog, and dbt.
Experience with Databricks Genie, MCP servers, or other approaches to providing governed data to AI agents.
Healthcare data experience, including claims (837/835), eligibility (834), FHIR, HL7 v2, or hospital event feeds, or experience in another regulated setting such as HIPAA or SOC 2.
Experience with workflow orchestration such as Databricks Workflows or Temporal, event-driven ingestion, and serverless compute.
Experience leading or managing engineers, or a clear interest in growing into a data engineering lead.

Responsibilities

Shape the data platform roadmap with data leadership, own delivery of work, and keep stakeholders informed about progress, risks, and trade-offs.
Set technical direction for data engineering, including design and code-review standards, mentoring, and unblocking engineers and contractors.
Own the Databricks and AWS platform architecture, including ingestion, storage, compute, access control, and environments; write design documents and make build-versus-buy decisions.
Build and maintain tested, versioned connectivity to partners and systems, including payer files over SFTP and S3, private networking, FHIR and HL7 clinical and event feeds, and data exchanged with the care platform.
Choose and optimize compute approaches, including serverless versus clusters, right-sizing, and migrating single-machine Python workloads to distributed compute.
Own orchestration and deployments, treating infrastructure and pipelines as code through Terraform, GitLab CI/CD, automated tests, and release practices.
Make the platform reliable and observable, detect failures and performance regressions early, and develop fixes and recovery mechanisms.
Make data usable by AI through governed metric definitions, Genie spaces, and MCP servers.
Validate incoming data against expectations, including row counts, date spans, required fields, and historical coverage.
Automate recurring manual work and routine data operations, including through AI agents.
Participate in a shared data operations rotation, kept light through automation.
Protect PHI through access controls, auditing, de-identification, and secure data exchange in partnership with security leadership.
Provide analytics engineering with a reliable, timely, cost-conscious platform on which to build the gold tier.
Represent data engineering with partners, vendors, and colleagues, and share learnings.