Principal Product Manager, Agentic Evals

Expedia•San Jose, WA

About The Position

Expedia Group is seeking a Principal Product Manager, Agentic Evals to join their Core AI Experiences team. This role is crucial for ensuring the quality, safety, and performance of AI-driven experiences across the company as AI becomes increasingly central to travel planning, booking, and management. The team is responsible for defining foundational methods for AI evaluation, including metrics, tooling, and processes. The Principal Product Manager will lead the strategy and development of Expedia Group’s AI evaluation framework, working closely with engineering, data science, and product teams to ensure consistency and evidence-based decision-making. This work will impact all AI-powered experiences and set company-wide standards for quality, safety, and trust.

Requirements

Bachelor's degree in a technical, quantitative, or related field, or equivalent practical experience.
10+ years of product management experience, including significant work with AI, ML, data platforms, or evaluation systems
Strong technical depth and experience partnering with DS/ML teams
Proven ability to build frameworks, metrics, or tools that scale across large organizations
Experience with model evaluation, instrumentation, experimentation, or system reliability
Strong written and verbal communication that brings clarity to complex technical spaces
Track record of influencing cross-functional leaders and driving alignment

Nice To Haves

Experience building evaluation platforms, ML tooling, or governance frameworks
Familiarity with LLM behavior, model evaluation methods, or prompt testing
Understanding of trustworthy AI principles, safety evaluation, and failure-mode analysis

Responsibilities

Define Expedia Group’s AI evaluation framework, principles, and adoption strategy
Build tools and workflows that support reliable, scalable measurement across brands
Partner with engineering and data science on metric design, instrumentation, and model evaluation
Establish clear, durable standards for accuracy, safety, transparency, and performance
Publish evaluation insights, scorecards, and recommendations that shape roadmaps
Influence strategic decisions on AI investments and risk management
Drive alignment across product, engineering, data science, and platform teams on evaluation strategy, standards, and prioritization
Create governance models that ensure consistency and responsible deployment at scale
Shape cross-team operating models and evaluation practices that guide how AI products are built and deployed across the company