Staff AI Product Analyst

Angi•Denver, CO

3d•$165,000 - $225,000•Remote

About The Position

The Staff AI Product Analyst will define AI-native product analytics for Angi, turning visionary technology into prioritized use cases, robust evaluation disciplines, and a trusted data ecosystem. This role is central to building Angi’s agentic product analytics capability—defining where AI-native analytics creates real leverage, designing the evaluation frameworks that earn organizational trust, and establishing repeatable system standards. As a senior individual contributor, you will have broad influence across the product analytics function and beyond, ensuring that both human teams and AI agents remain aligned on truth, quality, and usefulness. The ideal candidate is a systems-thinking data expert who bridges the gap between deep AI capabilities and practical business outcomes.

Requirements

10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment.
1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsibly.
Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices.
Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline.
Exceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholders.

Nice To Haves

Experience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systems.
Demonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iteration.
A systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business value.

Responsibilities

Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes.
Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship.
Map how agents interact with metrics, raw tables, documentation, and APIs to ensure requirements for data and analytics engineering are explicit, structured, and testable.
Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness.
Systematically categorize wrong answers, tool misuse, context gaps, and data edge cases, feeding those insights back into prompts, retrieval (RAG), tools, and data modeling pipelines.
Monitor live usage of Agentic AI analytics workflows—tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog.
Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Support enablement and leadership storytelling around agentic AI analytics.
Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomes.
Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metrics.