AI Agent Performance Engineer

Basata • Tempe, AZ

1d • Onsite

About The Position

We’re looking for an AI Agent Performance Engineer to own the quality, reliability, and performance of Basata’s conversational agents. You’ll build the observability, QA, and evaluation systems that ensure our agents consistently meet accuracy, speed, and reliability benchmarks. This role sits at the intersection of engineering, analytics, and applied AI. You’ll own and track the performance metrics that matter, detect drift or degradation early, and partner closely with Product and AI/ML engineering to drive measurable improvements in agentic behavior. Your work will directly shape how Basata delivers high-quality AI automation to every practice we support.

Requirements

• 3–5+ years of experience in software engineering and Applied AI.
• Hands-on experience building, deploying, and maintaining LLM-based agentic applications in production is essential.
• Strong working knowledge of agentic AI frameworks
• Experience building observability, monitoring, or QA systems using logs, metrics, dashboards, or automated evaluation pipelines.
• Proficiency with Python, analytics tooling , and comfort working with large-scale data logs.
• Experience in fast-paced, zero-to-one environments and comfort operating with autonomy.

Responsibilities

• Build and maintain observability and monitoring systems that track agent latency, accuracy, call outcomes, call-flow behavior, and failure modes in production.
• Develop rigorous QA and regression testing frameworks for Voice AI agents, including automated evaluation pipelines and scripted test scenarios.
• Define, instrument, and own key performance metrics and ensure agents consistently meet or exceed them.
• Analyze large volumes of agent interactions to identify trends, misclassifications, drift, prompt issues, or reliability gaps.
• Partner closely with Product, AI/ML, and Deployment teams to convert findings into concrete product, prompt, and architecture improvements .
• Build internal dashboards, alerts, and reports that make agent performance transparent for internal teams and customers.
• Conduct root-cause investigations for production issues and drive sustainable, long-term fixes.
• Document QA processes, performance baselines, and best practices to support fast scaling across specialties.

Benefits

• Drive real impact. Your work will help clinics run more smoothly and ensure patients get the care they need—faster and with fewer headaches.
• Shape something meaningful. From early product decisions to UI details, you'll play a big role in crafting both the code and the overall user experience.
• High ownership. Join a team where you’re trusted to lead, build, think critically, and bring ideas to life.
• Work with purpose. We’re not here to throw more tech at the wall, we're solving real problems in healthcare with tools that people rely on every day.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed