Software Engineer (AI Quality & Reliability)

Taxo•New York, NY

3d•$130,000 - $170,000•Onsite

About The Position

Taxo is building Voice AI agents to handle administrative work in healthcare, connecting with EHR and practice management systems for tasks like patient intake, scheduling, and refill requests. This role focuses on building automated evaluation and regression systems to ensure the AI's readiness for release. It's a software engineering position centered on test infrastructure, independent quality evidence, and reliable releases, aiming to make AI quality measurable in the context of healthcare workflows.

Requirements

Strong software engineering skills in Python or TypeScript.
Experience building maintainable test automation, reusable fixtures, and CI pipelines.
Practical experience evaluating LLM-powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring.
Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries.
A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage.
Clear communication about risk and release readiness.
Sound judgment around sensitive data and human review.

Nice To Haves

Experience with streaming audio.
Experience with telephony.
Experience with healthcare workflows.
Experience with EHR integrations.
Experience with reliability testing in privacy-sensitive products.

Responsibilities

Build shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or authorized, privacy-safe data.
Make evaluation runs repeatable and results easy to investigate.
Build automated end-to-end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation.
Evaluate task completion, tool-use correctness, latency, and conversation quality, combining deterministic checks with model-assisted scoring and human review.
Produce release-readiness evidence, maintain quality dashboards, and coordinate quality gates with product and engineering.
Surface regressions, coverage gaps, and unresolved risks before launch.
Reproduce defects, isolate failure patterns, and add durable regression cases.
Integrate automated checks into CI/CD, reduce flaky tests, and keep test environments and fixtures useful as the product evolves.
Partner with domain reviewers on healthcare-specific cases and escalate questions requiring clinical judgment.