Senior Data Platform Engineer
About The Position
The Senior Data Platform Engineer will design, build, operate, and improve the large-scale pipelines and foundational data platform behind Komodo Health’s Healthcare Map. This hands-on engineering role processes complex healthcare data at scale, improves system reliability and performance, and contributes to the technical direction of core data systems. The role is part of the Data Foundations team and helps create performant, trustworthy data assets for customer-facing products, internal innovation, and AI/ML use cases. The engineer will build and optimize batch pipelines, make storage and data-layout decisions, ingest data from external sources despite frequent schema changes, and transform healthcare claims, EHR, and reference data into serving-ready products. The role also strengthens quality, lineage, observability, and alerting, troubleshoots complex data and performance issues, and contributes to system design, testing, documentation, CI/CD, and rotational production support. The position is open to US remote work or hybrid work in San Francisco or New York City.
Requirements
- Strong hands-on experience building, owning, and debugging production data pipelines at multi-terabyte scale.
- Advanced Python skills, including writing performant, memory-efficient code for large-scale data processing, and strong SQL skills.
- Deep production experience with Airflow, including operating and scaling it, not just writing DAGs.
- Solid understanding of physical data storage, including columnar versus row formats and open table formats such as Iceberg, Delta Lake, or Hudi, including updates, compaction, small files, and partitioning.
- Experience with Spark or a comparable distributed processing framework.
- Proven experience designing and operating data systems in AWS.
- Strong data-correctness instincts, root-cause analysis skills, and production troubleshooting ability.
- Ability to communicate technical trade-offs clearly with engineering, product, and data partners.
- Comfort using AI-assisted engineering tools for coding, debugging, documentation, and technical exploration.
- Expected to use AI-augmented engineering tools to improve productivity and technical decision-making, including code generation and refinement, documentation, test creation, debugging, technical exploration, and evaluation of architectural trade-offs and risks.
Nice To Haves
- Experience with containers and orchestration, including Docker and Kubernetes.
- Strong core software engineering fundamentals and experience in more than one language.
- Data governance and lineage experience.
- Experience delivering external-facing data products through APIs, serving layers, or customer access patterns.
- Experience applying AI or agentic workflows to engineering, data quality, delivery, or operations.
- Experience succeeding in high-growth or ambiguous environments.
Responsibilities
- Build, operate, and optimize large-scale batch pipelines using Python, SQL, Spark, and Airflow on AWS.
- Make storage and data-layout decisions, including table formats, partitioning, file sizing, and compaction, to keep large datasets performant and cost-efficient.
- Ingest data from external sources and maintain pipeline correctness through frequent schema changes.
- Transform healthcare claims, EHR, and reference data into trusted, serving-ready data products for customers and internal teams.
- Strengthen reliability through data quality checks, validation, lineage, observability, and alerting.
- Debug complex data, system, and performance issues across compute-heavy workflows.
- Contribute to system design, code quality, testing, documentation, and CI/CD.
- Participate in rotational production support.
- Partner with Data Product and Engineering teams to deliver scalable, production-grade data solutions.
- Contribute to technical initiatives, design reviews, engineering best practices, and team mentorship.
Benefits
- Comprehensive health insurance.
- Dental insurance.
- Vision insurance.
- Flexible time off and holidays.
- 401(k) with company match.
- Disability insurance.
- Life insurance.
- Leaves of absence in accordance with applicable laws, regulations, and company policy.
- May be eligible for performance-based bonuses.
- May be eligible for equity awards.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
