Quality Lead, Agentic AI Workflow Evaluation

Innodata Inc.•San Jose, CA

3d•$75 - $85•Onsite

About The Position

Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide the data, evaluation frameworks, and human expertise needed to build trustworthy AI systems at scale. We offer solutions, platforms, and services for Generative AI/AI builders and adopters, leveraging our 36+ year legacy of delivering high-quality data and customer outcomes. This role is part of a dedicated onsite team evaluating complex, real-world agentic AI workflows for a frontier AI customer. Reviewers will work through ambiguous, multi-step scenarios in isolated test environments, assessing AI agent performance regarding safety, user intent, consent, and overall robustness. The Quality Lead is responsible for the quality of the output produced by the reviewers. This is a senior individual contributor role. While the Engagement Manager will manage the reviewers, the Quality Lead will set the quality standards, own the audit sample, run calibration sessions, maintain the rubric, and train reviewers. The Quality Lead will also serve as the deputy to the Engagement Manager, running the engagement in their absence. The quality approach for this engagement is not fully defined. The Quality Lead is expected to build this system in partnership with the customer's quality leads or, at minimum, to operate an existing system honestly and provide recommendations for improvement.

Requirements

Bachelor's degree or equivalent practical experience
4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain
Direct experience owning a quality function: you designed the audit, not just executed someone else’s
Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review
Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes
Demonstrated ability to run calibration with peers — including holding a position under disagreement and changing it when the argument is better
Strong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it
Comfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces)
Experience training or onboarding reviewers into rubric-based work

Responsibilities

Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported
Build the quality system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on data
Re-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes
Run calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases
Maintain rubric health — flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer
Train and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready
Give the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working
Report quality trends to the Engagement Manager and, alongside them, to the customer
Deputize for the Engagement Manager on delivery operations during absences
Maintain information security, privacy, and facility access practices required by the customer's onsite environment