AI Agentic Workflow Reviewer

Innodata • San Jose, CA

3d

About The Position

Innodata is seeking dedicated reviewers to evaluate complex, real-world agentic AI workflows. The role involves assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under scrutiny. This position requires judgment calls in ambiguous, multi-step scenarios, operating within isolated test environments, applying structured rubrics, and helping to define safe and policy-adherent agent behavior.

Requirements

• Bachelor's degree or equivalent practical experience
• 2–5+ years in quality review, QA/QC, trust & safety, content moderation, data annotation, or a similarly judgment-intensive review role
• Strong written communication – ability to justify a scoring decision clearly enough for someone else to audit it
• Comfort with ambiguity and structured decision-making under a rubric, rather than needing a fixed rulebook for every case
• Baseline understanding of AI/ML concepts and how AI agents complete tasks (can be learned through onboarding, but some familiarity helps)

Responsibilities

• Review multi-step agent task trajectories against rubric-based criteria covering task success, safety, and policy adherence.
• Evaluate whether an agent respected user agency and informed consent, not just whether it followed literal instructions.
• Apply privacy guardrails when reviewing tasks involving sensitive or personal data.
• Work inside isolated test environments, including verifying environment resets between test runs to prevent cross-contamination.
• Flag and escalate safety-relevant or ambiguous findings through defined escalation paths.

What This Job Offers

• Career Level: Mid Level