AI Agent Evaluation Engineer

Cresta • Canada (Remote), ON

10h

About The Position

Cresta is hiring its first AI Agent Evaluation Engineer to own the quality evaluation of its AI Receptionist, which handles millions of conversations each month across hundreds of customers. The role defines what quality means for the agent, builds systems to measure it across conversations, and provides a reliable assessment of how well the agent performs.

The work combines hands-on conversation review and annotation with evaluation design and engineering. The engineer will analyze real conversations to identify failure modes, build scalable evaluation systems—including LLM-based judges, golden sets, and regression suites—and calibrate automated evaluations against human judgment. The primary focus is producing an accurate, prioritized picture of agent quality; fixing identified issues is welcome but is not the core responsibility.

The role evaluates agent adherence, conversation outcomes, and user experience; covers edge cases and multilingual behavior; establishes quality and latency thresholds; and communicates findings to internal stakeholders and customers. It partners with engineering, product, and voice/language teams to turn evidence into prioritized recommendations and documents evaluation methods so they are reproducible.

Requirements

• At least 3 years of professional experience in software engineering, applied ML, or data-intensive research, including at least 1–2 years working hands-on with LLM-based systems in production.
• Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
• Experience building and operating an end-to-end evaluation system for a production LLM product, including a rubric, human labels, automated judge, and regression suite.
• Deep understanding of evaluation methodology, including the purposes and failure modes of golden sets, judges, regression suites, and calibration.
• Fluency in how LLM agents work, including prompts, tool calls, context handling, and multi-turn state, with the ability to interpret agent traces.
• Strong Python programming skills and experience with LLM APIs; ability to build and ship evaluation tooling independently.
• Ability to write rubrics that independent reviewers can score consistently and measure reviewer agreement.
• Ability to design tests from open-ended quality questions and determine what to measure and how.
• Willingness to listen to calls, label conversations, and build a failure taxonomy from scratch on a regular basis, personally and by directing QA partners.
• Professional fluency in at least one language other than English, sufficient to evaluate multilingual agent behavior directly.
• Strong written communication skills and ability to author guidelines others follow.

Responsibilities

• Define and maintain evaluation rubrics for agent adherence, outcomes, and conversation experience, including latency, turn-taking, tone, and repetition.
• Write annotation guidelines and a failure taxonomy, run calibration sessions, and measure inter-rater agreement.
• Classify and prioritize failure modes by severity and frequency.
• Review production conversations regularly, use QA partners, identify and cluster failures, and determine which evaluations should be automated.
• Trace failures to likely causes, such as prompts, knowledge gaps, tools or flows, and speech recognition, and route findings to the appropriate owners with supporting evidence.
• Evaluate behavior across languages and report quality by segments such as language, accent, audio quality, and intent.
• Build and operate an evaluation pipeline that scores production conversations using automated checks, LLM-based judges, and human review.
• Calibrate automated judges against human labels and monitor judge error.
• Design sampling strategies that account for intent, language, caller type, and audio conditions.
• Build and maintain golden sets and regression suites used to validate prompt, model, knowledge, and flow changes before release.
• Maintain a failure-mode coverage map and close evaluation gaps.
• Design evaluations for behaviors not covered by standard harnesses, including human handoffs, misuse and unwanted-caller handling, and resilience to upstream failures.
• Define quality and latency thresholds for the agent.
• Report quality trends clearly to internal stakeholders and customers, including uncertainty and unfavorable findings.
• Partner with forward-deployed engineers, backend engineers, product managers, and voice/language engineers to produce prioritized recommendations.
• Document evaluation methodology so results are reproducible and the practice can continue beyond the role holder.

Benefits

• Comprehensive medical, dental, and vision coverage with plans for employees and their families.
• Flexible paid time off.
• Paid parental leave for all new parents welcoming a new child.
• Retirement savings plan.
• Remote work setup budget for a productive home office.
• Monthly wellness and communication stipend.
• In-office meal program and commuter benefits for onsite employees.
• Equity as part of total compensation.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level