Data Analytics Lead

Happyrobot Inc.•San Francisco, CA

About The Position

HappyRobot is hiring a Data Analytics Lead to make data central to how the company builds and improves its products. The role partners closely with Product, Engineering, and Machine Learning teams to measure how model, agent, and product changes affect real-world performance. The successful candidate will define meaningful metrics, design experiments, and analyze how HappyRobot’s AI agents create value for clients. The work spans evaluating A/B tests and model changes, analyzing large volumes of conversations and workflows, and turning complex data into clear insights that inform product and ML roadmaps. The role also supports better instrumentation, data quality, experimentation systems, analytical data models, dashboards, and self-serve tools.

Requirements

At least 4 years of experience in Data Science, Product Analytics, or another highly quantitative product role.
Strong experience with experimental design, A/B testing, statistics, causal inference, and hypothesis-driven analysis.
Advanced proficiency in SQL and Python.
Experience defining and operationalizing product and feature metrics.
Ability to translate ambiguous product questions into rigorous analyses and actionable recommendations.
Strong product instincts and ability to distinguish statistical significance from meaningful product or customer impact.
Experience partnering closely with Product, Engineering, or Machine Learning teams.
Strong written and verbal communication skills.
High attention to detail and commitment to analytical accuracy.
Founder mindset, including ownership, independence, curiosity, and willingness to investigate deeply.

Nice To Haves

Experience with large language models, AI agents, generative AI, or other probabilistic ML products.
Experience measuring the production impact of model, prompt, retrieval, or orchestration changes.
Familiarity with ML evaluation systems and the relationship between offline evaluations and online metrics.
Experience analyzing conversational, NLP, speech, or other unstructured data.
Experience with enterprise or B2B products.
Experience combining quantitative analysis with qualitative methods such as conversation reviews, customer feedback, surveys, or user research.
Familiarity with modern analytics infrastructure, data warehouses, experimentation platforms, and business intelligence tools.
Prior experience in a fast-growing startup or another highly ambiguous environment.

Responsibilities

Define and track product-, feature-, and agent-level metrics.
Design, run, and interpret A/B tests for model changes, prompts, agent behavior, workflows, and product features.
Measure the effect of agent performance on client outcomes, including task completion, operational efficiency, response quality, and automation rates.
Connect offline model evaluations with production performance and real-world customer impact.
Analyze conversations, workflows, and product usage to identify opportunities and explain performance differences.
Investigate anomalies and regressions, conduct root-cause analyses, and recommend improvements.
Build statistical models, simulations, and analytical frameworks to support product and ML decisions.
Partner with Engineering to improve instrumentation, data quality, experimentation systems, and analytical data models.
Build dashboards and self-serve tools to help teams understand product and agent performance.
Communicate findings and recommendations clearly to technical and non-technical stakeholders.