Software Engineer, Voice Evaluation [NYC or SF]

Concurrence•San Francisco, CA
•$220,000 - $300,000•Onsite

About The Position

Concurrence is hiring a Software Engineer, Voice Evaluation to build and own the system that determines whether its voice agents are ready for patients. The role is responsible for realistic audio testing, scoring, and the release gate, including the authority to block releases that fail established criteria. Evaluations should detect failures that may not appear in transcripts, such as mishandled interruptions, missed clinical information, or failure to hand off to a person. The engineer will build simulated callers and call-replay tests that exercise the production voice path with real audio; test scenarios such as phone trees, hold queues, interruptions, conflicting clinical information, and escalation; develop and calibrate scoring systems; and assess whether simulation results predict behavior on real calls. The role also defines release criteria with model, runtime, product, and clinical colleagues, turns production failures into regression tests, verifies fixes, monitors post-release behavior, and maintains held-out evaluation sets. The position is based on site in New York City or San Francisco. Concurrence builds a platform for healthcare organizations to create clinical AI agents, backed by a Health World Model that unifies health-system data into an event-level patient-care record. The company reports that its agents have supported more than 10 million patient encounters and that it has about 35 people working in person in New York and San Francisco.

Requirements

  • Personally built and owned evaluation or simulation for production agents at scale.
  • Built agent simulations or replay systems, implemented scoring, and caught a concrete failure before release.
  • Evaluation results determined what shipped, and you led that work.
  • Can explain what a score captures and misses, and how it was checked against real behavior.
  • Writes reliable software that runs at volume and follows failures through to a verified fix.
  • Uses coding agents and independently verifies their output.
  • Low ego, direct, and able to uphold release criteria with evidence.
  • Able to work on site in New York City or San Francisco.

Nice To Haves

  • Voice and audio experience.
  • Experience calibrating automated judges against human labels.
  • Load testing and concurrent calls using audio.
  • Speech or audio machine learning experience.
  • Experience in healthcare or another regulated environment.

Responsibilities

  • Build simulated callers and call-replay tests that exercise the production voice path with real audio.
  • Test phone trees, hold queues, interruptions, conflicting clinical information, and appropriate escalation.
  • Build scoring systems and calibrate automated judges against human assessments.
  • Measure clinical failure rates and uncertainty, and test whether simulation results predict behavior on real calls.
  • Define release criteria with model, runtime, product, and clinical colleagues; own the release gate and block changes that fail the criteria.
  • Turn production failures into regression tests, verify fixes, and monitor behavior after release.
  • Maintain held-out evaluation sets so models are tested on examples they did not train against.

Benefits

  • Comprehensive health, dental, and vision insurance.
  • Daily catered lunch and dinner.
  • Mental health support and wellness coaching.
  • Flexible wellness stipend for fitness, therapy, or personal growth.
  • Annual learning budget for courses, books, or conferences.
  • Conference attendance budget for professional development.
  • Annual team offsite.
  • Academic collaboration opportunities.
  • Unlimited PTO.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service