Senior Applied AI Engineer (Observability)

Komodo Health • San Francisco, CA

8h • Hybrid

About The Position

At Komodo Health, our mission is to reduce the global burden of disease through smarter use of data. We built the Healthcare Map, the industry’s largest and most complete view of the U.S. healthcare system, by combining de-identified, real-world patient data with innovative algorithms and decades of clinical experience. This foundation supports a powerful suite of software applications, helping us answer healthcare’s most complex questions for our partners. Across the healthcare ecosystem, we help clients unlock critical insights to track detailed patient behaviors and treatment patterns, identify gaps in care, address unmet patient needs, and reduce the global burden of disease. We are grounded in our values: be awesome, seek growth, deliver “wow,” and enjoy the ride, joining a team of ambitious, supportive Dragons. Healthcare in the U.S. is complex, and Komodo Health is fixing that with data, having mapped the patient journey across the country to build the most complete picture of disease burden and treatment gaps. Our customers—pharma companies, payers, and health systems—use this data to make decisions that meaningfully improve patient outcomes. Labs@Komodo builds the AI-native platforms and systems that turn this data into action, including Marmot, Komodo’s AI-native product, designed with AI embedded directly into both the interface and the development workflow. By combining Komodo’s unmatched healthcare data with modern LLMs, Marmot is delivering some of the most compelling real-world insights of the new AI era. As a Senior Applied AI Engineer, you will own complex, full-stack AI solutions end-to-end—from applied research to production deployment. This role exists to set technical direction for ambiguous and high-impact use cases across Komodo, while scaling the AI systems, patterns, and infrastructure that enable reliable, repeatable delivery. You’ll mentor others, lead architectural decisions, and deepen Komodo’s AI-first culture. This role includes a specialization in AI observability, evaluation, and production reliability—ensuring systems are not only performant, but measurable, debuggable, and trustworthy in real-world use.

Requirements

• Proven track record of building end-to-end, production-grade AI systems.
• Expertise with LLMs, agent orchestration, multi-agent systems, and advanced prompt engineering.
• Strong fluency in Python and modern GenAI frameworks (vLLM, Crew AI, Strands, Chat Completions API).
• Full-stack depth enabling seamless integration of AI across front-end and back-end systems.
• Experience designing experiments, A/B tests, evaluation metrics, and performance instrumentation.
• Experience collaborating with platform/infrastructure teams on MLOps workflows.
• Strong cross-functional communication and mentorship capability.
• Hands-on experience with LLM evaluation techniques (offline and online), including defining quality metrics and benchmarking outputs.
• Experience monitoring and maintaining AI systems in production, including identifying and resolving failure modes (e.g., hallucinations, drift, tool failures).
• Drive experimentation across the organization, set best practices, and integrate new AI techniques into Komodo’s broader engineering ecosystem.

Nice To Haves

• Healthcare data expertise.
• Experience with distributed computing frameworks (e.g., Spark, Snowflake, Databricks) for large-scale data processing.
• Experience with observability or monitoring tooling (e.g., logging/tracing systems, ML monitoring platforms, or custom evaluation pipelines).

Responsibilities

• Shipped production-grade, full-stack AI solutions that materially enhance Komodo’s platform precision, reliability, or scalability.
• Led design and architecture for complex AI systems, including multi-agent orchestrations and advanced model pipelines.
• Prototyped and validated new applied research techniques, bringing academic insights into practical implementation.
• Designed A/B experiments and evaluation frameworks to measure AI impact in production.
• Established robust observability frameworks for AI systems, including monitoring, alerting, and failure analysis across LLM-driven workflows.
• Mentored engineers across teams in prompt engineering, debugging, agent orchestration, and AI system design.
• Influenced MLOps pipeline improvements (model versioning, automated monitoring, CI/CD for AI).
• Architecting, building, and deploying end-to-end AI systems that balance innovation with reliability and ethical considerations.
• Leading solution design for ambiguous AI problems across Komodo’s platform and internal operations.
• Collaborating with product, data, and platform teams to define requirements and shape strategic AI investments.
• Designing advanced prompt chains, multi-agent flows, and complex evaluation frameworks.
• Driving applied research by experimenting with cutting-edge models, techniques, and academic papers.
• Contributing to internal AI standards, reusable templates, and high-performance orchestration patterns.
• Transitioning prototypes into scalable systems with comprehensive observability, alerting, and governance.
• Defining and implementing metrics, logging, and tracing strategies for LLM systems (e.g., hallucination detection, tool usage, latency, and output quality).
• Investigating and debugging production AI failures, including model drift, degraded outputs, and agent/tool breakdowns.

Benefits

• Comprehensive health, dental, and vision insurance
• Flexible time off and holidays
• 401(k) with company match
• Disability insurance
• Life insurance
• Leaves of absence in accordance with applicable state and local laws and regulations and company policy.
• Performance-based bonuses
• Equity awards

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed
• Number of Employees: 251-500 employees