Lead Product Data Scientist, Agents

Klaviyo•Boston, MA

About The Position

Klaviyo is seeking a Lead Product Data Scientist to own and advance the measurement and data science strategy for Composer and Customer Agent. This highly visible, hands-on technical leadership role combines technical expertise, product and business judgment, and strategic influence across AI product development. The role defines how Klaviyo measures agent adoption, engagement, quality, cost, and business impact, and establishes analytical foundations for understanding customer value. The successful candidate will develop rigorous approaches to evaluating agent performance and determining whether AI-powered experiences improve customer outcomes. The work combines experimentation, causal inference, and advanced analytics to inform decisions in an area where measurement frameworks are still evolving. The role partners with Product, Engineering, Finance, and executive leadership on product strategy, pricing, monetization, and investment decisions; defines shared metrics, methodologies, and data foundations; and mentors data scientists. The team takes an AI-first approach, using coding agents, LLMs, and other emerging tools while maintaining rigorous validation and quality standards.

Requirements

10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine-learning, or LLM-powered products in production.
Deep expertise in product measurement, experimentation, statistical modeling, and causal inference, including selecting and defending rigorous approaches when randomized experiments are not feasible.
Experience evaluating AI systems in production, including designing evaluation frameworks, analyzing agent interactions or conversations, using human labeling or LLM-as-judge methods, and understanding how model, prompt, and tool changes affect outcomes.
High proficiency in SQL and Python or R, with experience developing production-quality analytical solutions and familiarity with dbt and modern analytics engineering practices.
An AI-first approach to work, actively using coding agents, LLMs, and emerging tools while understanding their limitations and ensuring output accuracy and reliability.
Demonstrated technical leadership in complex analytical initiatives, including setting direction, mentoring data scientists, reviewing methodologies, and influencing technical standards across teams.
Statistical rigor and product and business intuition, including an understanding of how customer engagement, AI product performance, usage-based pricing, and unit economics relate to customer and business outcomes.
Ability to solve ambiguous, emerging problems and develop practical frameworks that balance analytical rigor with business needs.
Track record of influencing senior stakeholders, aligning cross-functional teams around shared definitions and priorities, and translating complex findings for technical and non-technical audiences.
Strong communication skills, including explaining sophisticated analytical concepts, constructively challenging assumptions, and communicating uncertainty and trade-offs.
Ability to thrive in fast-moving product environments, adapt to evolving priorities, and balance analytical depth with the urgency and iteration required for AI-powered products.
Degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering, or another quantitative discipline, or an MBA with a quantitative emphasis; equivalent hands-on applied data science experience is equally valued.

Nice To Haves

Experience measuring and optimizing AI agents, conversational AI, or LLM-powered products at a high-growth B2B SaaS or product-focused technology company.
Hands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize, including agent traces, tool-call data, and evaluation pipelines.
Experience with usage-based or consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics.
Experience building governed metrics, semantic layers, metric trees, or standardized measurement frameworks used across multiple product teams.
Experience applying classification, clustering, or other advanced analytical methods to large volumes of unstructured text, customer conversations, or agent interactions.
Experience developing internal AI agents or AI-powered analytics tools that improve team productivity and decision-making.
Familiarity with marketing attribution, incremental measurement, or evaluating the impact of AI-driven customer engagement.
Background in marketing technology, ecommerce, CRM, customer-service software, or other data-driven SaaS products.

Responsibilities

Own end-to-end agent measurement for Composer and Customer Agent, connecting adoption, engagement, performance, customer value, and revenue impact.
Partner with Product and Engineering to define agent effectiveness measures, including task completion, customer outcomes, failure rates, and quality, cost, and business-value trade-offs.
Build scalable evaluation frameworks using conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification.
Measure causal impact and incremental value using experimentation and advanced statistical approaches, including difference-in-differences, matched comparisons, staged rollouts, and synthetic controls.
Partner with Finance and Product to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and the financial implications of product investments.
Define consistent measurement methodologies, governed data models, and evaluation practices that support confident decision-making as AI products and technologies evolve.
Use coding agents, LLMs, and emerging tools to accelerate SQL development, data modeling, dashboard creation, conversation classification, and analytical workflows, while validating outputs and maintaining quality.
Build scalable, self-service analytics capabilities so product managers and cross-functional stakeholders can explore agent performance, evaluate customer outcomes, and identify improvement opportunities.
Partner with Engineering and Analytics Engineering to develop reliable instrumentation, data pipelines, and dbt models for agent interactions, evaluation, and data quality.
Lead the technical direction of agent analytics by reviewing analytical approaches, mentoring data scientists, and establishing best practices for statistical rigor, measurement quality, and AI-native work.
Translate complex findings into clear, actionable recommendations; align cross-functional stakeholders on meaningful metrics; and communicate uncertainty and trade-offs effectively.
Evaluate emerging measurement methodologies, challenge assumptions, and prioritize analytical questions as Klaviyo's agents and AI strategy evolve.