QA Engineer, Voice AI Agents
About The Position
Arbiter is the AI-powered care orchestration system that unites healthcare. We are launching our best-in-class, patient-facing Agentic platform to optimize patient outcomes through a unique multimodal approach. We optimize complex healthcare workflows that interface with patients using the latest Agentic AI approaches, and we combine it with a sophisticated platform to serve this Agentic layer at scale. We are looking for expert engineers and leads to join our team and help us push the frontier of what's possible with Agentic workflows in Healthcare. Backed by one of the largest seed rounds in health tech history and operators who bring the expertise and distribution to scale nationally, we're building the connected infrastructure healthcare should have had all along.
Requirements
- 3 to 5 years in QA, support engineering, technical operations, or a similar role on a production software platform.
- A strong debugging instinct. You like finding out why something broke, and you don't stop at the first plausible answer.
- Comfort reading logs, querying data with SQL, and working with APIs and JSON or YAML configuration.
- Experience writing test plans and bug reports that engineers trust.
- Clear written communication and precise documentation habits.
- Good judgment on severity and prioritization when several issues are live at once.
Nice To Haves
- Experience with voice AI, conversational AI, IVR, or contact center platforms.
- Scripting ability in Python or similar for automating checks and analysis.
- Exposure to LLM evaluation, prompt testing, or conversation quality scoring.
- Healthcare experience, especially payer or provider outreach, and familiarity with HIPAA and PHI handling.
- Experience with observability tools or with call analytics tools.
Responsibilities
- Investigate reported and detected issues on live campaigns, including dropped calls, wrong agent responses, failed handoffs, incorrect scheduling outcomes, and data mismatches.
- Trace problems across call recordings, transcripts, system logs, campaign configuration, and integration data to find the actual root cause.
- Write clear, reproducible bug reports that engineers can act on right away, with severity, member impact, and supporting evidence.
- Triage incoming issues, separating agent behavior from configuration errors, data problems, and platform bugs, and route each to the right owner.
- Run pre-launch QA on new campaigns and workflow changes: test scripts, edge cases, escalation paths, and handoffs to nurses or live staff.
- Carry out ongoing conversation QA on live campaigns against defined quality standards, and flag trends before they turn into customer issues.
- Check that campaigns behave as designed: correct targeting, sequencing, disposition logic, and outcome capture.
- Help build and maintain the QA framework: test cases, scoring rubrics, regression suites, and release checklists.
- Automate repetitive checks where it makes sense, such as transcript analysis, config validation, and outcome reconciliation.
- Track quality metrics and report patterns to deployment, product, and engineering so issues get fixed at the source.
Benefits
- Highly Competitive Salary & Equity Package: Designed to rival top FAANG compensation, including meaningful equity.
- Generous Paid Time Off (PTO): To ensure a healthy work-life balance.
- Comprehensive Health, Vision, and Dental Insurance: Robust coverage for you and your family.
- Life and Disability Insurance: Providing financial security.
- Simple IRA Matching: To support your long-term financial goals.
- Professional Development Budget: Support for conferences, courses, and certifications to fuel your continuous learning.
- Wellness Programs: Initiatives to support your physical and mental health.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
