Senior Software Engineer, Data Platform

Carrum Health•,

9h•$160,000 - $180,000

About The Position

The Data Operations team at Carrum Health is responsible for the entire data supply chain, including ingesting eligibility and claims data from numerous insurance carriers and data providers. This data is processed into reliable datasets that support member experiences, clinical outcomes, and business reporting, and is delivered to partners and internal teams with high accuracy and timeliness. Currently, much of this work is operational, with each new client launch, data provider switch, or recurring claims load consuming engineering resources. This role aims to build a robust data platform with production services, well-defined contracts, infrastructure as code, and observability as code, transforming carrier-specific tasks into configurable processes on a solid foundation. The team is AWS-native, utilizing services like ECS Fargate, Batch, S3, RDS, Athena, Lambda, Step Functions, EventBridge, and Transfer Family, with dbt as the transformation backbone and Datadog for observability. The engineering culture is AI-forward, with engineers using AI tooling daily, including AWS Bedrock (Claude) for job monitoring and failure summarization. A key aspect of this role will be extending this AI foundation to develop systems for AI-assisted ingestion, classification, and transformation, further automating carrier-specific work.

Requirements

5+ years of professional software engineering experience with a track record of owning production systems end to end.
Strong Python and SQL skills for batch jobs, Lambda functions, and data transformation via dbt (Trino/Athena dialect).
Production-grade software engineering practices, including strict testing discipline with comprehensive automated test coverage (unit, integration, contract) on all new code.
CI/CD comfort (GitHub Actions, CircleCI, or comparable).
Deep AWS experience with ECS, Batch, Lambda, S3, RDS, Athena, Step Functions, EventBridge, Transfer Family, IAM, Secrets Manager, and CloudWatch, including operating and debugging production systems on this stack.
Terraform fluency, including designing or extending Terraform modules, understanding environment promotion, secret management, and IAM patterns.
dbt proficiency in modeling, testing, and documentation.
A platform engineering mindset, focusing on systems, contracts, and reusable abstractions.
Strong debugging and incident response instincts, with a focus on fixing systemic issues.
High bias for action and self-unblocking, with a proven track record of proactive cross-team communication.
Daily AI usage in engineering workflow, including using AI tools for code generation, system design, debugging, and documentation.
Ability to maintain rigorous technical documentation, including project decisions, technical notes, and blocker histories in Jira.
Experience writing Technical Design Documents before implementing large projects.

Nice To Haves

LLM/RAG experience, including familiarity with LLM APIs, RAG architectures, vector stores (pgvector, FAISS), or agentic frameworks (LangChain, LlamaIndex).
Experience with Databricks or comparable lakehouse platforms (Delta Lake, Spark, Iceberg).
PostgreSQL depth, including data modeling, indexing, query performance, and understanding application-layer data flows.
Experience with streaming or CDC architectures (Kafka, Kinesis, Flink, Debezium, or similar).
Experience in healthcare, insurance, or another highly regulated industry, including familiarity with claims data formats (837P/I), eligibility files (834/820), and HIPAA compliance.
Familiarity with SFTP-based data exchange, PGP encryption, and integrating with external data providers.
Experience with data quality frameworks (dbt tests, Great Expectations, Monte Carlo).
Comfort reading and navigating Ruby/Rake to debug and extend the current data pipeline during migration.

Responsibilities

Own and extend the data pipeline platform, driving its evolution by adding automation, testing, structured observability, and configuration-driven abstractions to reduce the cost of new integrations. Assist in the long-term migration towards Python and Spark-based compute.
Build production platform services using Python and SQL for batch jobs, Lambda functions, and data services that ingest, transform, validate, and route data, ensuring the output is production-ready software with tests, contracts, and monitoring.
Develop platform tooling that creates leverage for the team, such as code generators for eligibility automation, integration testing infrastructure, classification Lambdas, and data-fix utilities, focusing on systemic solutions rather than one-off fixes.
Treat infrastructure as a first-class engineering discipline by designing and extending Terraform modules, building environment promotion, secret management, IAM patterns, and deployment pipelines, and migrating operational configurations from runbooks into Terraform-managed resources.
Build observability into the platform, including structured JSON logging, Datadog monitors and log pipelines, SLOs, data quality checks (e.g., EWMA control charts), and automated alerting, integrated at the architectural level.
Leverage AI as a force multiplier by extending the existing foundation of AI tools (like Bedrock/Claude) into ingestion tooling, data classification, and transformation assistance, actively using AI for code generation, system design, debugging, and documentation.
Eliminate operational toil through software development, addressing operational load such as recurring eligibility and claims loads, SFTP setup, PGP key management, and data provider switches by building systemic solutions.
Drive platform modernization by evaluating and integrating Databricks, and making architectural decisions related to storage layout, compute models, governance, and migration paths.
Raise the engineering bar by defining patterns for testing, code review, CI/CD, deployment, and incident response, documenting architectural decisions, and writing maintainable code.
Partner across the organization with analytics engineers, data scientists, software engineers, and client-facing teams to translate requirements into platform capabilities.
Learn the healthcare data domain, including medical claims, insurance eligibility, and HIPAA, to build an operationally and clinically correct platform.