Lead Product Data Scientist, Agents

Klaviyo • Boston, MA

4d • $172,000 - $258,000

About The Position

Klaviyo is seeking a Lead Product Data Scientist to own and advance measurement and data science strategy across Composer and Customer Agent. This hands-on technical leadership role combines data science, AI product development, and business strategy to define how the company measures agent adoption, engagement, quality, cost, and business impact.

The role develops rigorous frameworks for evaluating agent performance and determining whether AI-powered experiences improve customer outcomes. It applies experimentation, causal inference, and advanced analytics to address emerging measurement challenges, and partners with Product, Engineering, Finance, and executive leadership on product strategy, pricing, monetization, and investment decisions.

The Lead Product Data Scientist also shapes the technical direction of agent analytics, mentors data scientists, and establishes shared metrics, methodologies, and data foundations. The position uses an AI-first approach to analytics, including coding agents, LLMs, and emerging tools, while maintaining rigorous validation and quality standards. The role may require up to 10% travel, coordinated in advance.

Requirements

• 10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production.
• Deep expertise in product measurement, experimentation, statistical modeling, and causal inference, including selecting rigorous approaches when randomized experiments are not feasible.
• Experience evaluating AI systems in production, including designing evaluation frameworks, analyzing agent interactions or conversations, using human labeling or LLM-as-judge methodologies, and understanding the effects of model, prompt, and tool changes.
• High proficiency in SQL and Python or R, with experience developing production-quality analytical solutions and familiarity with dbt and modern analytics engineering practices.
• Ability to use coding agents, LLMs, and emerging tools to accelerate analytical workflows while understanding their limitations and ensuring output accuracy and reliability.
• Technical leadership experience setting direction on complex analytical initiatives, mentoring data scientists, reviewing methodologies, and influencing technical standards.
• Statistical rigor and product and business intuition, including understanding customer engagement, AI product performance, usage-based pricing, and unit economics.
• Ability to develop practical frameworks for ambiguous problems where established metrics or methodologies may not exist.
• Track record of influencing senior stakeholders, aligning cross-functional teams, and translating complex findings into clear recommendations for technical and non-technical audiences.
• Strong communication skills, including explaining sophisticated analytical concepts, constructively challenging assumptions, and communicating uncertainty and trade-offs.
• Ability to thrive in fast-moving product environments, adapt to changing priorities, and balance analytical depth with product iteration.
• Degree in Statistics, Mathematics, Computer Science, Economics, Finance, Engineering, or another quantitative discipline, or an MBA with a quantitative emphasis; equivalent hands-on applied data science experience is equally valued.

Nice To Haves

• Experience measuring and optimizing AI agents, conversational AI, or LLM-powered products in a high-growth B2B SaaS or product-focused technology company.
• Hands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize, including agent traces, tool-call data, and evaluation pipelines.
• Experience with usage-based or consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics.
• Experience building governed metrics, semantic layers, metric trees, or standardized measurement frameworks used across multiple product teams.
• Experience applying classification, clustering, or other advanced analytical methods to large volumes of unstructured text, customer conversations, or agent interactions.
• Experience developing internal AI agents or AI-powered analytics tools that improve team productivity and decision-making.
• Familiarity with marketing attribution, incremental measurement, or evaluating the impact of AI-driven customer engagement.
• Background in marketing technology, ecommerce, CRM, customer-service software, or other data-driven SaaS products.

Responsibilities

• Own agent measurement across Composer and Customer Agent, connecting adoption, engagement, agent performance, customer value, and revenue impact.
• Partner with Product and Engineering to define agent effectiveness, task completion, customer outcomes, failure rates, and trade-offs among quality, cost, and business value.
• Build scalable agent evaluation frameworks using conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification.
• Measure causal impact and incremental value using experimentation and advanced statistical methods, including quasi-experimental approaches.
• Partner with Finance and Product to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and product investment implications.
• Establish trusted metrics, consistent measurement methodologies, governed data models, and evaluation practices.
• Use coding agents, LLMs, and emerging tools to accelerate analytics workflows while maintaining validation and quality standards.
• Build scalable self-service analytics capabilities for product managers and cross-functional stakeholders.
• Partner with Engineering and Analytics Engineering to develop instrumentation, data pipelines, and dbt models that capture agent interactions and support reliable evaluation.
• Lead the technical direction of agent analytics, review analytical approaches, mentor data scientists, and establish best practices.
• Translate complex analytical findings into actionable recommendations, align stakeholders on metrics, and communicate uncertainty and trade-offs.
• Evaluate emerging methodologies and prioritize analytical questions as Klaviyo's agents and AI strategy evolve.

Benefits

• The total compensation package may include participation in the company's annual cash bonus plan.
• Equity may be included in the total compensation package.
• Sign-on payments may be included in the total compensation package.
• A comprehensive range of health, welfare, and wellbeing benefits is available based on eligibility.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior