Data Analyst II - Agentic Experiences

Expedia•Seattle, WA

5d•$112,000 - $179,000•Hybrid

About The Position

Expedia Group brands power global travel for everyone, everywhere. We design cutting-edge tech to make travel smoother and more memorable, and we create groundbreaking solutions for our partners. Our diverse, vibrant, and welcoming community is essential in driving our success. Our Technology Team partners with teams across Expedia Group to create innovative products, services, and tools to deliver high-quality experiences for travelers, partners, and our employees. A singular technology platform powered by data and machine learning provides secure, differentiated, and personalized experiences that drive loyalty and traveler satisfaction. We are looking to gain an established performer who consistently applies analytical capabilities to design, run, and interpret evaluations for agentic AI systems, with a moderate level of guidance and direction. Regularly interacts with stakeholders up to Senior Manager level, including ML scientists, product teams, and other evaluation partners.

Requirements

1+ years for Bachelors/Masters grad (preference for a Mathematics, Scientific, Linguistics, or Cognitive Science degree) OR 2+ years in a comparable data analytics, AI evaluation, or annotation role with relevant experience
Demonstrated experience delivering data-driven insights and recommendations that drove change or performance improvement for stakeholders
Hands-on experience with data annotation or labeling workflows, including quality control and rubric design

Nice To Haves

Familiarity with agentic AI systems, LLMs, or AI product evaluation is strongly preferred

Responsibilities

Designs and recommends evaluation approaches appropriate to the agentic task — including human eval, automated metrics, and hybrid methods
Understands the difference between a statistically valid eval readout and exploratory analysis; applies this distinction when reporting results to stakeholders
Frames business or product questions as concrete evaluation tasks, breaking them into manageable steps with clear success criteria
Selects appropriate measurement techniques for agentic systems, including turn-level and task-level assessments, and understands the caveats of each
Identifies failure modes in agent behavior (e.g. hallucination, tool misuse, incorrect task completion) and surfaces these clearly in eval outputs
Favors iterative delivery — works with stakeholders to refine eval scope, agree on what is in scope for each phase, and evolves methodology based on findings
Designs and documents annotation rubrics and guidelines that produce consistent, high-quality labels at scale
Performs and quality-checks annotations, identifying inter-annotator disagreement and proposing rubric improvements to address ambiguity
Has hands-on experience using annotation platforms and can onboard peers or vendors onto annotation tasks with minimal support
Understands how annotation quality directly impacts model and eval reliability, and proactively flags risks
Extracts and combines data from multiple sources to support eval pipelines and annotation workflows
Writes intermediate SQL and Python scripts in a shareable, reproducible way; stores and versions work in tools like GitHub or Confluence
Understands data quality best practices and applies them to eval datasets, including checks for label consistency, coverage, and bias
Has a working knowledge of the most important data sources relevant to their work area and knows how to unblock data issues
With manager support, clearly articulates eval goals, methodology, caveats, and conclusions to both technical and non-technical audiences
Creates clear, audience-appropriate artifacts — technical documentation, eval summaries, and executive-facing write-ups — and shares them in the right forums
Builds trust and collaborates transparently with stakeholders; seeks peer reviews and actively contributes to team upskilling
Seeks feedback early and demonstrates the ability to act on it across current and future projects
Demonstrates a proactive approach to identifying problems, surfacing blockers, and becoming increasingly self-sufficient over time