Senior Manager, Business Operations eCommerce

Walmart • Bentonville, AR

13d • Onsite

About The Position

Position Summary... What you'll do... Key Responsibilities AI Evaluation & Transparency Manage dashboards connecting capability metrics with business metrics to surface what’s working and what isn’t. Maintain golden datasets for benchmarking model relevance, execution accuracy, personalization, and compliance. Partner with AI Tech and Tech Ops to calibrate LLM-as-Judge and ensure automated evaluation remains aligned with human judgment. Shopper Experience Quality & Audits Conduct periodic audits of shopper-facing AI experiences (answers, nudges, recommendations, in-shop chat). Define and monitor KPIs for intent recognition, execution accuracy, and personalization precision. Run cross-journey evaluations to identify friction or divergence patterns. SOP Governance & Continuous Improvement Own and evolve SOPs for annotation, labeling, and evaluation to reflect changing features and customer contexts. Capture and codify new edge cases into updated labeling guides. Ensure all SOPs align with shopper-first outcomes and compliance . Human + AI Divergence Monitoring Track variance between human labels, automated judgments, and live model outputs. Create structured escalation paths for high-risk failures (irrelevant, unsafe, or biased responses). Quantify divergence trends and drive remediation with Product and Engineering. Vendor & Partner Management Manage external labeling partners, ensuring throughput, accuracy, and compliance SLAs. Deliver training grounded in real Sparky interactions and golden dataset examples. Audit vendor output and hold partners accountable for quality metrics. Closed-Loop Feedback & Model Integration Build and manage the continuous feedback loop that connects: Shopper signals Human review & annotation Automated LLM evaluation Ensure that every signal from real interactions feeds directly into model training, experience design, and KPI improvement —driving measurable gains in personalization, nudge effectiveness, and conversational trust

Requirements

• 2–4 years in AI operations, ML evaluation, or eCommerce data quality ; experience managing annotation or human-in-the-loop programs preferred.
• Working knowledge of AI evaluation frameworks (LLM-as-Judge, golden datasets, A/B testing).
• Skilled in KPI design, audit processes, and vendor oversight .
• Analytical storyteller able to link model metrics to business outcomes.
• Obsessed with customer trust, transparency, and continuous improvement.
• Bachelor's degree in information technology, computer science, or related area and 5 years’ experience in eCommerce merchandising, site operations, business management, or related area OR 7 years’ experience in eCommerce merchandising, site operations, business management, information technology, computer science or related area.
• 2 years’ supervisory experience.

Nice To Haves

• ECommerce Merchandising, Site Operations, Business Management, or related area
• Master's degree in Information Technology, Computer Science, or related area

Responsibilities

• AI Evaluation & Transparency Manage dashboards connecting capability metrics with business metrics to surface what’s working and what isn’t.
• Maintain golden datasets for benchmarking model relevance, execution accuracy, personalization, and compliance.
• Partner with AI Tech and Tech Ops to calibrate LLM-as-Judge and ensure automated evaluation remains aligned with human judgment.
• Shopper Experience Quality & Audits Conduct periodic audits of shopper-facing AI experiences (answers, nudges, recommendations, in-shop chat).
• Define and monitor KPIs for intent recognition, execution accuracy, and personalization precision.
• Run cross-journey evaluations to identify friction or divergence patterns.
• SOP Governance & Continuous Improvement Own and evolve SOPs for annotation, labeling, and evaluation to reflect changing features and customer contexts.
• Capture and codify new edge cases into updated labeling guides.
• Ensure all SOPs align with shopper-first outcomes and compliance .
• Human + AI Divergence Monitoring Track variance between human labels, automated judgments, and live model outputs.
• Create structured escalation paths for high-risk failures (irrelevant, unsafe, or biased responses).
• Quantify divergence trends and drive remediation with Product and Engineering.
• Vendor & Partner Management Manage external labeling partners, ensuring throughput, accuracy, and compliance SLAs.
• Deliver training grounded in real Sparky interactions and golden dataset examples.
• Audit vendor output and hold partners accountable for quality metrics.
• Closed-Loop Feedback & Model Integration Build and manage the continuous feedback loop that connects: Shopper signals Human review & annotation Automated LLM evaluation Ensure that every signal from real interactions feeds directly into model training, experience design, and KPI improvement —driving measurable gains in personalization, nudge effectiveness, and conversational trust

Benefits

• At Walmart, we offer competitive pay as well as performance-based bonus awards and other great benefits for a happier mind, body, and wallet.
• Health benefits include medical, vision and dental coverage.
• Financial benefits include 401(k), stock purchase and company-paid life insurance.
• Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty, and voting.
• Other benefits include short-term and long-term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more.
• You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes.
• The amount you receive depends on your job classification and length of employment.
• It will meet or exceed the requirements of paid sick leave laws, where applicable.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees