(USA) Principal, Data Scientist | Gen AI Vision

Walmart • Sunnyvale, CA

2d

About The Position

Overview Walmart’s Next Gen Commerce team is building the future of conversational shopping with intelligent agents that reason, recommend, and proactively assist customers. As a Principal Data Scientist for Quality & LLM Judging Systems, you will serve as the technical lead for defining and measuring the success of these AI systems. You will be responsible for designing the "brain" that critiques our agents, utilizing a mix of LLM-as-a-judge frameworks, human benchmarks, and automated pipelines. In this high-impact, hands-on role, you will partner closely with engineering and product leaders to translate subjective quality goals into rigorous, actionable metrics that drive model improvement and safe deployment.

Requirements

• Education: Advanced degree (Master's or PhD) in Computer Science, Statistics, Mathematics, Computational Linguistics, or a related field.
• Experience: 7+ years of experience in Data Science or Machine Learning with a focus on NLP, Deep Learning, or AI evaluation.
• Generative AI Expertise: Deep understanding of Large Language Models (LLMs), including prompt engineering, chain-of-thought reasoning, and instruction tuning.
• Technical Proficiency: Solid understanding of Python and expertise with core data science packages (NumPy, Pandas, PyTorch, Scikit-learn).
• Metric Design: Proven experience designing metrics for non-deterministic outputs (e.g., evaluating summarization, relevance, or helpfulness).
• Engineering Fundamentals: Experience building scalable data pipelines and familiarity with distributed training/inference frameworks.
• Option 1: Bachelors degree in Statistics, Economics, Analytics, Mathematics, Computer Science, Information Technology or related field and 5 years' experience in an analytics related field.
• Option 2: Masters degree in Statistics, Economics, Analytics, Mathematics, Computer Science, Information Technology or related field and 3 years' experience in an analytics related field.
• Option 3: 7 years' experience in an analytics or related field

Nice To Haves

• PhD in Machine Learning, NLP, or a related quantitative field.
• Experience with conversational AI, chatbots, summarization, retrieval-augmented generation, or recommendation evaluation in an e-commerce context.
• Knowledge of model distillation, LoRA, instruction tuning, or parameter-efficient adaptation techniques
• Familiarity with evaluating open-ended outputs where ground truth is subjective or contextual
• Publications, patents, or open-source contributions in LLM evaluation or applied AI
• Data science, machine learning, optimization models, PhD in Machine Learning, Computer Science, Information Technology, Operations Research, Statistics, Applied Mathematics, Econometrics
• Publications or active peer reviewer in related journals or conference
• Successful completion of one or more assessments in Python, Spark, Scala, or R
• Using open source frameworks (for example, scikit learn, tensorflow, torch)
• We value candidates with a background in creating inclusive digital experiences, demonstrating knowledge in implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly.
• The ideal candidate would have knowledge of accessibility best practices and join us as we continue to create accessible products and services following Walmart’s accessibility standards and guidelines for supporting an inclusive culture.

Responsibilities

• Develop Evaluation Architectures: Design and implement state-of-the-art evaluation pipelines for conversational agents using LLM-as-a-judge, and hybrid scoring frameworks.
• Prompt Engineering & Calibration: Develop high-precision prompts for evaluator models and rigorously test them against human judgment to ensure high inter-rater reliability.
• Model Distillation & Optimization: Lead the fine-tuning of smaller, cost-effective models to act as scalable "Judge" models, balancing trade-offs between accuracy, latency, and cost.
• Dataset Curation: Work with large-scale conversation logs to curate "Golden Set" datasets and design annotation instructions that standardize ground truth for subjective tasks.
• Cross-Functional Integration: Collaborate with Engineering teams to integrate quality signals into CI/CD pipelines, enabling automated regression testing and production monitoring.
• Failure Mode Analysis: Conduct deep-dive analyses on agent failures (hallucinations, tool misuse, safety violations) and define actionable feedback loops for the modeling team.
• Insight Discovery & Strategic Influence: Leverage evaluation data to discover systemic weaknesses and root causes, actively influencing sub-agent modeling teams and cross-functional partners to prioritize and drive targeted improvements in overall performance.
• Thought Leadership: Mentor senior data scientists, standardize best practices for evaluation across the org, and maintain world-class credentials through patents, publications, or conference presentations.

Benefits

• At Walmart, we offer competitive pay as well as performance-based bonus awards and other great benefits for a happier mind, body, and wallet. Health benefits include medical, vision and dental coverage. Financial benefits include 401(k), stock purchase and company-paid life insurance. Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty, and voting. Other benefits include short-term and long-term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more. You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes. The amount you receive depends on your job classification and length of employment. It will meet or exceed the requirements of paid sick leave laws, where applicable.
• Live Better U is a Walmart-paid education benefit program for full-time and part-time associates in Walmart and Sam's Club facilities. Programs range from high school completion to bachelor's degrees, including English Language Learning and short-form certificates. Tuition, books, and fees are completely paid for by Walmart.
• Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Number of Employees: 5,001-10,000 employees