Senior Applied AI Engineer (Observability)

Komodo Health•San Francisco, CA

23h•$200,000 - $270,000•Hybrid

About The Position

At Komodo Health, the mission is to reduce the global burden of disease through smarter use of data. They built the Healthcare Map, the industry’s largest and most complete view of the U.S. healthcare system, by combining de-identified patient data with innovative algorithms and clinical experience. This map powers a suite of software applications that help partners answer complex healthcare questions, track patient behaviors, identify care gaps, address unmet needs, and reduce disease burden. Komodo Health values being awesome, seeking growth, delivering 'wow,' and enjoying the ride, fostering a team of ambitious, supportive individuals passionate about their mission. The U.S. healthcare system is fragmented and inefficient, and Komodo Health aims to fix this with data, enabling life sciences companies, payers, and providers to make better decisions that improve patient outcomes. To fully leverage this data, Komodo is investing in AI-native infrastructure to make AI systems reliable, scalable, and deeply embedded in product development and usage. This role is for a Senior Applied AI Engineer to join a new AI Platform / Observability team. The focus is on building the foundation for trustworthy AI systems at scale, in a backend-heavy, systems-focused, greenfield environment. The engineer will own problems end-to-end, from exploration to production, and help define how AI systems are built, evaluated, and operated across Komodo. The initial focus is on AI observability, evaluation, and production reliability, with scope expanding to broader platform ownership including agent systems, orchestration layers, and shared infrastructure. Komodo Health emphasizes that AI is foundational to their work and expects every team member to integrate AI into their daily tasks to drive efficiency and success.

Requirements

Experience building production AI systems end-to-end (not just prototypes)
Strong expertise with LLMs and prompt systems
Strong expertise with agent orchestration and tool/function calling
Hands-on experience with AI observability, evaluation, or monitoring systems
Hands-on experience with debugging and improving production AI behavior
Strong backend engineering skills: Python, APIs, distributed systems, or platform architecture
Experience designing evaluation frameworks and experiments (A/B testing, benchmarking)
Ability to operate in ambiguous, fast-moving environments
Strong communication and mentorship skills

Nice To Haves

Healthcare data expertise
Experience with distributed computing frameworks (e.g., Spark, Snowflake, Databricks) for large-scale data processing
Experience building internal observability platforms
Experience with LLM evaluation or monitoring systems
Familiarity with request tracing, replay systems, or model diagnostics

Responsibilities

Build the observability and reliability foundation for AI systems across Komodo (logging, tracing, evaluation pipelines, feedback loops)
Define how the organization measures LLM performance and quality in production (hallucinations, drift, latency, failure modes)
Ship production-grade AI systems that improve platform reliability, scalability, and performance
Lead design and architecture for complex applied AI systems (multi-agent workflows, tool-calling systems, model pipelines)
Establish evaluation frameworks and experimentation practices (A/B testing, offline + online evaluation)
Contribute to reusable infrastructure, patterns, and standards adopted across teams
Design and implement logging, tracing, and request visibility for LLM systems
Design and implement evaluation pipelines and benchmarking frameworks
Design and implement feedback loops for continuous system improvement
Define metrics for output quality and correctness
Define metrics for latency and system performance
Define metrics for tool usage and agent behavior
Detect and debug hallucinations, model drift and system degradation and failure modes
Architect and deploy end-to-end AI systems agent-based workflows, prompt chains and tool integrations and scalable LLM-powered services
Transition prototypes into reliable, production-grade systems
Contribute to shared AI infrastructure and orchestration patterns
Partner with product, data, and platform teams to shape AI-driven solutions
Drive experimentation across the organization
Set best practices
Integrate new AI techniques into Komodo’s broader engineering ecosystem
Integrate AI into daily work from summarizing documents to automating workflows and uncovering insights

Benefits

Comprehensive health insurance
Dental insurance
Vision insurance
Flexible time off
Paid holidays
401(k) with company match
Disability insurance
Life insurance
Leaves of absence in accordance with applicable state and local laws and regulations and company policy
Performance-based bonuses
Equity awards