AI Operations Analyst

FICO

7d•$105,000 - $165,000

About The Position

The AI Operations Analyst will support myFICO’s expanding AI-powered product features within the Scores B2C team. The role reviews AI chat conversations and Personalized Credit Insights summaries to identify inaccurate, unsupported, off-topic, or unsafe responses, then uses those findings to improve product quality. The analyst also monitors member feedback and quality trends, maintains labeled regression data, and reports findings and recommended fixes. The role works with Product, the Applied AI Engineer, and QA engineers on review tooling, rubrics, and sampling methods, and verifies fixes after releases. It also coordinates escalations with Legal, Compliance, and Trust & Safety, following flagged items through resolution. The position requires sound judgment, consistent application of written review criteria, strong SQL and dashboard skills, clear written communication, and discretion when handling sensitive consumer data in a regulated environment.

Requirements

Bachelor's degree in a related field or equivalent experience.
Demonstrated experience in a quality, analytical, or operations role reviewing customer conversations, such as CX or QA, support operations, compliance QA, or conversation design.
Strong SQL skills and comfort building and reading dashboards in a BI tool such as Tableau.
Sound judgment and close attention to detail, with the ability to apply a written rubric consistently and explain decisions.
Strong written communication, including clear failure descriptions and concise summaries for leadership.
Comfort working with sensitive consumer data in a regulated environment and the discretion that requires.
Experience working in Agile/Scrum teams.

Nice To Haves

Experience in consumer finance, credit, banking, fintech, or another regulated environment, including FCRA, ECOA, or UDAAP.
Knowledge of credit reports and FICO Scores.
Experience with conversational AI or LLM-based products.
Experience with data labeling, rubric design, inter-rater agreement, or failure-mode taxonomies.
Basic Python and Jupyter experience.
Experience with Legal and Compliance escalation processes.

Responsibilities

Review a risk-weighted sample of AI chat conversations and Personalized Credit Insights summaries, flagging inaccurate, unsupported, off-topic, or unsafe responses.
Categorize member feedback, guardrail blocks, and escalations into a failure-mode taxonomy, and track changes over time.
Own and maintain the golden regression dataset by adding labeled examples from production misses, keeping labels consistent, and documenting the rubric.
Label data to measure and improve automated evaluators, including double-labeling a share of items to measure reviewer agreement.
Publish a weekly quality report covering intent coverage, resolution and containment, refusal and guardrail rates, safety flags, top failure modes, and recommended fixes.
Escalate flagged content to Legal, Compliance, and Trust & Safety with clear severity ratings, and follow each item through resolution.
Use SQL and dashboards to analyze member questions, experience gaps, and quality drift.
Partner with Product, the Applied AI Engineer, and QA engineers on review tooling, rubrics, and sampling methods, and verify fixes after each release.