Senior Backend Engineer, AI Evaluations

Ellipsis Health•San Francisco, CA

About The Position

The Senior Backend Engineer will join the AI Evaluation team and build infrastructure and internal tooling for reliable, repeatable evaluation of AI systems in production. The role includes developing API and data integrations, test configuration management, and reporting tools that make evaluations easy to run, extend, and operate at scale. The engineer will also build and maintain evaluation frameworks for system components such as ASR, LLMs, TTS, knowledge bases, and guardrails, and productionalize them for regular analysis, regression detection, and continuous monitoring. The role also involves creating debugging tools to inspect end-to-end calls, trace failures, and surface relevant signals so internal teams can diagnose issues effectively. The ideal candidate is a pragmatic, infrastructure-minded engineer who turns ad hoc analysis into durable systems, values developer experience and software quality, and builds tooling that makes complex AI systems observable, testable, and easier to operate.

Requirements

At least 5 years of professional software engineering experience, with a strong focus on backend systems, platforms, or developer tooling.
Proven experience designing and maintaining production-grade infrastructure with code, including APIs, services, and data pipelines.
Strong proficiency in at least one general-purpose programming language, such as Python, TypeScript/JavaScript, or Java.
Experience using test automation frameworks, evaluation pipelines, or CI/CD-integrated testing systems.
Familiarity with observability and debugging tools, including logging, metrics, and tracing, and with building internal tools that improve developer and QA workflows.
Strong debugging skills and a methodical approach to diagnosing production and evaluation issues.
Ability to collaborate effectively across engineering, QA, and operations teams and translate requirements into reliable, maintainable systems.
Product-minded approach to infrastructure, with attention to usability, documentation, and long-term maintainability.

Nice To Haves

Experience working with complex, multi-component systems such as ASR, LLMs, TTS, or other ML-powered services.
Experience working in healthcare or other regulated environments, including awareness of HIPAA and PHI handling.
Familiarity with conversational AI or voice agents, including multi-turn dialogue, latency constraints, and error recovery.
Familiarity with LLM observability or evaluation tools, such as Langfuse or prompt evaluation frameworks.
Background in digital health, care coordination, or patient-facing systems.

Responsibilities

Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams, including an automated testing platform for AI voice agents and debugging and observability tools.
Develop and productionalize evaluation frameworks for individual system components such as ASR, LLMs, TTS, knowledge bases, and guardrails.
Partner with ML, engineering, and QA teams to translate evaluation requirements into robust, maintainable infrastructure and tooling.
Improve developer experience by making evaluation systems easy to extend, well-documented, and reliable in day-to-day use.
Ensure evaluation tooling meets production standards for reliability, performance, and maintainability.