Principal Product Manager, Agentic Evals

Expedia Group•San Jose, CA

5d•$224,000 - $387,000•Onsite

About The Position

At Expedia Group, we help travelers explore the world, one journey at a time. As a global travel company powered by passionate people, trusted partnerships, and leading technology, we connect travelers, partners, and advertisers through our consumer brands, B2B network, and travel advertising business. Here, you'll do meaningful work that helps millions of people discover, book, and experience travel with more ease, confidence, and joy. Our five Behaviors-Traveler First, Think Big, Operate with Excellence, Ownership Mindset, and Succeed Together-help foster a supportive environment where people can grow their careers and have the flexibility, benefits, and support to do their best work. Join us and build for travelers everywhere. Introduction to The Team: The Product Team creates high-quality end-to-end experiences for travelers, partners, and Expedia Group. Our focus on customer-centric innovation enables us to develop products that build loyalty and repeat business. We partner closely with teams across Expedia Group to drive growth and achieve results for our customers and the company. Expedia Group powers travel for everyone, everywhere. As AI becomes core to how travelers plan, book, and manage their trips, our ability to evaluate quality, safety, and performance becomes essential. You will build the frameworks and tools that ensure every AI experience meets a high standard of clarity, accuracy, and trust. The Core AI Experiences team defines the foundational methods for evaluating AI across the company. We create the metrics, tooling, and processes that guide how teams measure accuracy, safety, latency, helpfulness, and long-term traveler outcomes. In this role, you will lead the strategy and build-out of Expedia Group’s AI evaluation framework. You will work closely with engineering, data science, and product teams to drive consistency and support decisions grounded in evidence. Your work will influence every AI-powered experience across Expedia Group and set the bar for quality, safety, and trust company-wide.

Requirements

Bachelor's degree in a technical, quantitative, or related field, or equivalent practical experience.
10+ years of product management experience, including significant work with AI, ML, data platforms, or evaluation systems
Strong technical depth and experience partnering with DS/ML teams
Proven ability to build frameworks, metrics, or tools that scale across large organizations
Experience with model evaluation, instrumentation, experimentation, or system reliability
Strong written and verbal communication that brings clarity to complex technical spaces
Track record of influencing cross-functional leaders and driving alignment

Nice To Haves

Experience building evaluation platforms, ML tooling, or governance frameworks
Familiarity with LLM behavior, model evaluation methods, or prompt testing
Understanding of trustworthy AI principles, safety evaluation, and failure-mode analysis

Responsibilities

Define Expedia Group’s AI evaluation framework, principles, and adoption strategy
Build tools and workflows that support reliable, scalable measurement across brands
Partner with engineering and data science on metric design, instrumentation, and model evaluation
Establish clear, durable standards for accuracy, safety, transparency, and performance
Publish evaluation insights, scorecards, and recommendations that shape roadmaps
Influence strategic decisions on AI investments and risk management
Drive alignment across product, engineering, data science, and platform teams on evaluation strategy, standards, and prioritization
Create governance models that ensure consistency and responsible deployment at scale
Shape cross-team operating models and evaluation practices that guide how AI products are built and deployed across the company