AI Reliability Manager

Thomson Reuters•Eagan, MN

2d•Hybrid

About The Position

The Risk & Fraud Product Management team is looking for an AI Reliability Manager to help ensure the agentic AI capabilities in CLEAR, our investigative platform, are accurate, trustworthy, and continuously improving. Our customers include law enforcement, financial crimes compliance teams, corporate fraud investigators, and government agencies who use our products to make consequential decisions. The quality of what our AI produces matters enormously, and this role exists to measure that quality, defend it, and drive it upward. The AI Reliability Manager is an individual contributor who is an analytically rigorous, resourceful problem-solver who is genuinely curious about how agentic AI systems work and how to make them better. You and the reliability team will own the evaluation data and quality signals that tell us whether our AI is performing to standard, translate customer-reported issues into actionable engineering work, and serve as a trusted voice on release readiness. This is a hands-on role for someone who likes digging into messy output, finding the pattern in it, and turning that pattern into tangible improvements. You will join an established evaluation program with existing gold datasets, annotation guidelines, and scoring rubrics already in production use, and you will be responsible for extending and scaling that work as our AI capabilities expand.

Requirements

Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work
Experience applying consistent standards to open-ended work where reasonable people can disagree about what "good" looks like
Hands-on use of generative AI tools, with enough curiosity to have noticed how they fail, including confident answers that aren't supported and sources that don't say what the AI claims
Strong written communication, including explaining technical findings to non-technical audiences

Nice To Haves

Experience in one of our primary customer segments: law enforcement or criminal investigations, AML/BSA or financial crimes compliance, corporate fraud or corporate security, or government program integrity.
Working understanding of how retrieval-based AI systems generate and cite sources, and of common failure modes including hallucination, citation mismatch, and incomplete retrieval.

Responsibilities

Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing.
Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short.
Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused.
Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness.
Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day-to-day experts, ensuring they apply guidelines consistently.
Serve as a day-to-day product resource for the sales channel, answering questions about product behavior, capabilities, and known limitations, and keeping them current on the status of open issues.
Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely.
Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners.
Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one-off tickets.
Partner with product management, engineering, applied research, and go-to-market teams to translate quality findings into roadmap and prioritization decisions.
Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows.

Benefits

Hybrid Work Model
Flexibility & Work-Life Balance
Work from anywhere for up to 8 weeks per year
Career Development and Growth
Continuous learning and skill development
Grow My Way programming
Skills-first approach
Flexible vacation
Two company-wide Mental Health Days off
Access to the Headspace app
Retirement savings
Tuition reimbursement
Employee incentive programs
Resources for mental, physical, and financial wellbeing
Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more.
Two paid volunteer days off annually
Opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives.
Market competitive health, dental, vision, disability, and life insurance programs
Competitive 401k plan with company match
Competitive vacation, sick and safe paid time off
Paid holidays (including two company mental health days off)
Parental leave
Sabbatical leave
Optional hospital, accident and sickness insurance paid 100% by the employee
Optional life and AD&D insurance paid 100% by the employee
Flexible Spending and Health Savings Accounts
Fitness reimbursement
Access to Employee Assistance Program
Group Legal Identity Theft Protection benefit paid 100% by employee
Access to 529 Plan
Commuter benefits
Adoption & Surrogacy Assistance
Access to Employee Stock Purchase Plan