Senior Machine Learning Engineer

Ambience Healthcare • San Francisco, CA

1d • $225,000 - $300,000 • Hybrid

About The Position

As a Senior Machine Learning Engineer at Ambience, you will build and improve the AI systems that power our clinical products. You’ll own complex projects end-to-end, from diagnosing production failures and designing evaluations to building, deploying, and iterating on model and agentic systems. This is a highly hands-on role with significant technical ownership. You’ll work closely with clinicians, product managers, and fellow engineers to translate cutting-edge research into reliable, production-grade AI systems.

Requirements

• 5+ years in production ML, research engineering, or applied AI.
• Have built a consequential production AI system or materially improved model behavior in production.
• Strong understanding of modern LLMs, transformers, and production AI systems.
• Experienced designing evaluations for LLMs, agents, or other complex AI systems.
• Can turn ambiguous quality problems into measurable dimensions, datasets, and experiments.
• Familiar with challenges such as grader bias, leakage, misleading aggregate metrics, regression detection, and offline-online mismatch.
• Experience building production systems involving multiple models, tools, retrieval, context, state, routing, or orchestration.
• Understands reliability and failure modes in complex AI workflows, not just individual model calls.
• Proficient in Python and modern ML frameworks; PyTorch preferred.
• Comfortable with deployment, observability, CI/CD, and containerized systems.
• Still highly hands-on: writes code, inspects traces, analyzes failures, and debugs production systems.
• Skilled at building high-quality datasets and feedback loops.
• Experienced using production failures, user feedback, and active learning to improve model and system quality.
• Able to work closely with clinicians, product managers, and fellow engineers.
• Strong communicator who can simplify complex AI concepts for diverse audiences.
• Comfortable owning ambiguous technical problems and driving them to measurable outcomes.

Nice To Haves

• Experience with realtime voice, conversational AI, or multimodal systems.
• Experience with fine-tuning, post-training, or model adaptation.
• Prior work in healthcare, clinical AI, or other regulated, high-stakes industries.
• Experience interviewing or mentoring ML engineers.
• Open-source contributions to ML, agent, or evaluation tooling.

Responsibilities

• Build Trustworthy AI Evaluation Systems: Design and own evaluation pipelines for LLM and agentic systems, combining automated graders, regression testing, production feedback, and human evaluation to measure real product quality.
• Improve Production Model Behavior: Diagnose high-impact failure modes and test improvements across prompting, retrieval, context, routing, data, fine-tuning, or other model and system interventions.
• Build Agentic AI Systems: Develop production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
• Build Data and Improvement Flywheels: Turn production failures and user feedback into better datasets, evaluations, and model behavior through active learning and systematic iteration.
• Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, NLP, speech, and multimodal AI and translate promising ideas into practical experiments.
• Own AI Systems End-to-End: Work across models, data, evaluation, orchestration, serving, and observability, while remaining deeply hands-on in code and production debugging.

Benefits

• Comprehensive medical, dental, and vision coverage for you and your dependents
• 401(k) with a company match of up to 3% of base salary
• A remote-friendly culture (with a San Francisco HQ) and full equipment provisioning to ensure you can work effectively from wherever you’re based.
• Parental leave to support your family needs
• Annual company-wide off-sites, team off-sites and regular team lunches and all-hands gatherings, with travel, lodging and meals covered
• Flexible time off with no annual cap, company-wide holidays and an annual holiday shutdown from December 24–January 1 designed to support real rest and long-term sustainability.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed