AI Quality Engineer/Operator

Recruiting From Scratch•San Francisco, CA

6d•Onsite

About The Position

Our client is building enterprise-grade AI automation systems for healthcare operations, focused on accelerating patient access to critical treatments and specialty medications. Their platform automates workflows including benefit verification, prior authorization, patient onboarding, financial assistance enrollment, and adherence management using AI agents deployed in real enterprise healthcare environments. This is an opportunity to join a rapidly scaling AI healthcare startup at an early stage, where the team has already secured major enterprise contracts and is operating at the frontier of agentic AI systems in regulated healthcare environments. The company is highly execution-focused, profitable, and building foundational infrastructure for enterprise AI operations in healthcare.

Requirements

0–7 years of experience in QA engineering, operations, legal, technical program management, AI operations, or related precision-oriented roles
Strong written communication and analytical thinking skills
Experience working with AI systems, LLMs, prompt engineering, or conversational AI
Comfortable working inside IDEs and technical tooling environments
Ability to learn lightweight scripting and debugging workflows quickly
Strong attention to detail and operational rigor
Experience debugging complex workflows or systems
Ability to think systematically about root causes and edge cases
Comfortable operating in ambiguous startup environments
Strong learning mindset and technical curiosity
Able to work autonomously with minimal process structure
Interest in AI systems, healthcare operations, or enterprise automation

Nice To Haves

Experience with LLMs, AI agents, prompt engineering, or AI evaluations
QA engineering or quality operations experience
Strong written communication and documentation ability
Experience with regex, scripting, or lightweight automation
Healthcare, biopharma, or regulated-industry experience
Operations-heavy startup background
Legal, compliance, or analytical writing backgrounds
Strong debugging and root-cause analysis mindset
Experience reviewing AI outputs at scale
Technical curiosity combined with operational rigor
Comfortable learning new tools independently
Ability to critically evaluate AI-generated outputs
High attention to detail under pressure
Strong execution orientation

Responsibilities

Review and debug AI agent outputs across transcripts, workflows, recordings, and operational systems
Identify quality issues, edge cases, and failure patterns in production AI systems
Write, refine, and test prompts for conversational AI and agent workflows
Run systematic root-cause analysis and quality investigations
Document issues rigorously with clear categorization, reproduction steps, and escalation context
Build and maintain lightweight scripts, regex patterns, and automation workflows using AI-assisted coding tools
Operate continuous quality improvement loops for production AI agents
Validate fixes and ensure issue resolution across workflows end-to-end
Collaborate closely with engineers, operations, customer teams, and leadership
Work directly inside IDEs and AI tooling environments
Evaluate LLM outputs critically rather than relying blindly on AI-generated responses
Improve operational reliability and quality processes for enterprise healthcare AI systems
Contribute to the development of scalable AI quality infrastructure
Operate in a highly ambiguous, fast-moving startup environment with strong ownership expectations