Engagement Manager, Agentic AI Workflow Evaluations
About The Position
Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence. We provide the data, evaluation frameworks, and human expertise needed to build trustworthy AI systems at scale. We offer a range of solutions, platforms, and services for Generative AI/AI builders and adopters, leveraging our 36+ year legacy of delivering high-quality data and outstanding customer outcomes. This role involves standing up and managing a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. The Engagement Manager will lead a team of ten reviewers and one QA lead working within isolated test environments. The team's primary focus is assessing whether AI agents complete tasks safely, respect user intent and consent, and withstand scrutiny. This is a delivery leadership position with a significant customer-facing component, requiring the manager to be the primary day-to-day contact at the site. Accountability includes throughput, quality, and turnaround time, with a responsibility to identify opportunities for engagement growth. While not performing daily scoring tasks, the role demands sufficient technical depth to audit the team's work and defend scoring decisions to the customer's technical leads.
Requirements
- Bachelor's degree or equivalent practical experience
- 5+ years of professional experience, including direct management of a delivery team of 10–20 people
- Track record as the customer-facing owner of a services or delivery engagement with an enterprise or technology client, including metrics reporting and issue escalation
- Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review
- Ability to read rubric-scored work critically and form an independent view of whether a score is correct
- Strong written communication; able to produce reporting and escalation memos that a technical customer will accept without rework
Responsibilities
- Own end-to-end delivery for the onsite team: throughput, quality, turnaround, and capacity against committed volumes
- Manage and develop a team of eleven, including hiring, onboarding, performance management, and retention
- Serve as the primary onsite point of contact for the customer's program and technical leads; run recurring reviews, report on delivery metrics, and resolve issues before they escalate
- Audit reviewer output directly — sample trajectories, check rubric application, and assess whether scoring rationale would survive customer review
- Partner with the QA lead on calibration cycles, quality metrics and evaluation, and rubric refinement; arbitrate disagreements that calibration does not resolve
- Own the escalation path for safety-relevant and ambiguous findings, including judgment on what reaches the customer and how quickly
- Identify expansion opportunities within the account: adjacent workstreams, new task types, additional capacity, and scope the work with internal delivery and commercial teams
- Forecast staffing and cost against the engagement's commercial model, and flag variance early
- Maintain information security, privacy, and facility access practices required by the customer's onsite environment
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
