Senior Product Manager, AI Evaluation & Quality| Level 18| US: Remote

Circana•,

1d•$150,000 - $159,000•Remote

About The Position

The Liquid AI Evals role will build and own the evaluation framework used to validate the quality, reliability, safety, and business value of Circana's AI agents. Reporting to the Liquid AI Product Leader, this role will establish repeatable testing practices that help the team move quickly while ensuring new agent capabilities meet client, business, and technical expectations before and after launch. The role defines evaluation strategies, methodologies, metrics, and acceptance thresholds; builds frameworks for AI agent and agentic workflow capabilities; and partners with product and engineering teams to validate agent performance. It develops representative tests, datasets, benchmarks, and scoring approaches, evaluates agent performance and client value, identifies failure patterns, monitors production traces, documents results, and promotes consistent evaluation standards across product verticals.

Requirements

4–6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field.
Understanding of statistical evaluation, testing methodologies, and performance metrics.
Familiarity with evaluating AI agents or LLM-based systems, including techniques such as LLM-as-a-judge scoring and production trace analysis.
Ability to design structured tests for complex or ambiguous use cases.
Strong analytical reasoning and attention to detail.
Comfort operating in an early-stage, fast-moving environment with evolving requirements.
Ability to communicate technical findings clearly to varied audiences.
Bachelor’s degree in a quantitative, technical, or related discipline, or equivalent experience.

Responsibilities

Define evaluation strategies, methodologies, metrics, and acceptance thresholds for Circana's AI agents.
Build evaluation frameworks for AI agent and agentic workflow capabilities.
Partner with the Liquid AI Product Leader, engineering, and each product vertical to validate agent performance before release.
Develop representative test scenarios, golden datasets, benchmarks, and scoring approaches, including LLM-as-a-judge methods where appropriate.
Evaluate agent quality, accuracy, consistency, reliability, safety, and client value.
Identify failure patterns and translate findings into product improvement recommendations.
Establish ongoing monitoring of production traces to detect performance changes following launch.
Document evaluation results and communicate findings to technical and business audiences.
Promote consistent evaluation standards as Liquid AI's capabilities expand across product verticals.