Principal Product Manager, Agentic Evals

Expedia•San Jose, CA

2d•$224,000 - $338,500

About The Position

Expedia Group is hiring a Principal Product Manager to lead the strategy and build-out of its company-wide AI evaluation framework. The role sits on the Core AI Experiences team, which defines foundational methods for evaluating AI and creates the metrics, tooling, and processes teams use to measure accuracy, safety, latency, helpfulness, and long-term traveler outcomes. The Product Manager will build frameworks, tools, and workflows that support reliable, scalable evaluation across Expedia Group’s brands. Working closely with engineering, data science, and product teams, this person will establish standards for AI quality, safety, transparency, and performance; publish evaluation insights and recommendations; and influence AI investment and risk-management decisions. The position also drives cross-functional alignment, governance, and operating models for responsible AI development and deployment across the company.

Requirements

Bachelor’s degree in a technical, quantitative, or related field, or equivalent practical experience.
10+ years of product management experience, including significant work with AI, ML, data platforms, or evaluation systems.
Strong technical depth and experience partnering with data science and machine learning teams.
Proven ability to build frameworks, metrics, or tools that scale across large organizations.
Experience with model evaluation, instrumentation, experimentation, or system reliability.
Strong written and verbal communication skills, with the ability to bring clarity to complex technical spaces.
Track record of influencing cross-functional leaders and driving alignment.

Nice To Haves

Experience building evaluation platforms, ML tooling, or governance frameworks.
Familiarity with LLM behavior, model evaluation methods, or prompt testing.
Understanding of trustworthy AI principles, safety evaluation, and failure-mode analysis.

Responsibilities

Define Expedia Group’s AI evaluation framework, principles, and adoption strategy.
Build tools and workflows that support reliable, scalable measurement across brands.
Partner with engineering and data science on metric design, instrumentation, and model evaluation.
Establish clear, durable standards for accuracy, safety, transparency, and performance.
Publish evaluation insights, scorecards, and recommendations that shape roadmaps.
Influence strategic decisions on AI investments and risk management.
Drive alignment across product, engineering, data science, and platform teams on evaluation strategy, standards, and prioritization.
Create governance models that ensure consistency and responsible deployment at scale.
Shape cross-team operating models and evaluation practices that guide how AI products are built and deployed across the company.