AI Quality Associate

Accordion•Boston, MA

14h•Hybrid

About The Position

The AI Quality Associate focuses on validating the quality of AI-generated outputs, agent behaviors, and AI-assisted workflows. This impactful role will be critical in building benchmark scenarios, defining scoring rubrics, evaluating business usefulness, and identifying failure patterns that conventional pass or fail software testing would not catch. This position will ideally be based in one of our US offices and is a hybrid role with the flexibility to work remotely 2 days per week (also eligible for remote candidates). This position is not eligible for immigration sponsorship.

Requirements

4+ years of experience in QA, analytics, business process validation, AI evaluation, operations, or similar roles
Strong writing, analysis, and pattern-recognition skills
Experience evaluating outputs against nuanced criteria rather than only binary correctness
Ability to work with structured rubrics, scenario libraries, and evidence-based reviews – you are thoughtful, precise, and highly discerning
Ability to spot subtle output problems others miss
Comfort with ambiguity but disciplined in scoring and documentation
Comfort collaborating across Engineering and business teams, with a focus on trust, usefulness, and business reality
Bachelor's degree preferred

Nice To Haves

Experience with finance, accounting, FP&A, transaction services, or business process design preferred
Excited to be part of a growing team, with a focus on driving future growth
Passionate about delivering exceptional client service
Knowledgeable about the relationship between Private Equity sponsors and their portfolio companies, particularly within the office of the CFO
Willing to get your hands dirty in the details of a project while simultaneously seeing the whole picture
Comfortable managing projects with multiple complex workstreams while also focused on your single workstream (you are managing and doing)
A self-starter with a strong work ethic
Full of entrepreneurial spirit and comfortable in a fluid, flat organization
Able to effectively communicate complex issues and solutions, and raise issues to senior team members when necessary
A team player, able to work with team members across all levels
Someone who enjoys mentoring others and doing meaningful work
A leader of others; you lead by example

Responsibilities

Design and execute structured evaluations for AI-enabled features and workflows
Assess outputs for groundedness, instruction adherence, consistency, usefulness, tone, control compliance, and risk
Identify hallucinations, unsupported assertions, missing logic, and unsafe recommendations
Build and maintain golden datasets, benchmark prompts, comparison sets, and scorecards
Develop rubrics that allow quality to be measured consistently across releases and changes
Compare performance across prompt versions, workflow revisions, tools, and models
Support release decisions with evidence on quality regression or improvement
Work closely with Finance SMEs, product managers, and engineers to determine what acceptable looks like in real business contexts
Help define human-review thresholds and escalation patterns for higher-risk use cases
Analyze reviewer feedback, override patterns, and live quality signals to improve evaluation coverage over time