AI Agentic Workflow Reviewer

Innodata Inc. • San Jose, CA

3d • $50 - $55

About The Position

Innodata is seeking reviewers to evaluate complex, real-world agentic AI workflows. The role involves assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under scrutiny. This is not routine content moderation; reviewers will handle ambiguous, multi-step scenarios requiring judgment calls. The work will be performed in isolated test environments, applying structured rubrics to help define safe and policy-adherent agent behavior.

Requirements

• Bachelor's degree or equivalent practical experience
• 2–5+ years in quality review, QA/QC, trust & safety, content moderation, data annotation, or a similarly judgment-intensive review role
• Strong written communication – ability to justify a scoring decision clearly enough for someone else to audit it
• Comfort with ambiguity and structured decision-making under a rubric, rather than needing a fixed rulebook for every case
• Baseline understanding of AI/ML concepts and how AI agents complete tasks (can be learned through onboarding, but some familiarity helps)

Responsibilities

• Review multi-step agent task trajectories against rubric-based criteria covering task success, safety, and policy adherence.
• Evaluate whether an agent respected user agency and informed consent, not just whether it followed literal instructions.
• Apply privacy guardrails when reviewing tasks involving sensitive or personal data.
• Work inside isolated test environments, including verifying environment resets between test runs to prevent cross-contamination.
• Flag and escalate safety-relevant or ambiguous findings through defined escalation paths.

What This Job Offers

• Career Level: Mid Level