Principal, Data Scientist, Agentic AI Systems Engineering & Model Post-Training

Walmart•Bentonville, AR

3d•$110,000 - $220,000•Onsite

About The Position

Walmart’s Supply Chain AI Lab & Innovation Factory is building a new generation of production-grade agentic AI systems that reason over complex enterprise information, coordinate specialized agents, use tools safely, plan and execute long-horizon work, and continuously improve through rigorous evaluation and model post-training. This role exists to build those systems end to end—and to improve the models that power them. This is not an analytics-focused data science role. It is a deeply hands-on AI systems engineering position focused on designing, building, and operating production software. As a Principal Data Scientist in this space, you are a hands-on technical leader. You quickly turn hard, ambiguous problems into working full-stack prototypes—with a real user experience, APIs, telemetry, and an evaluation plan—then harden them into secure, reliable, observable, maintainable production systems. You independently own major product and platform domains within the shared agentic architecture, across the full technology stack—agent orchestration and model logic, backend services and APIs, the data layer, and web and CLI/TUI interfaces. These capabilities will first be applied to some of Walmart's most complex operational and supply-chain problems, beginning with the Autonomous Supply Chain Engine and its Discovery Loop—a continuous, human-in-the-loop mechanism that monitors signals, discovers opportunities and risks, reasons through changing conditions, and generates strategic recommendations. Build and own key capabilities of the Discovery Loop within our Autonomous Supply Chain Engine. This is a flagship capability for this role: you will own major parts of its execution model, knowledge and context layer, evaluation loop, and learning path that turns feedback and business outcomes into measurable improvement. Prior supply chain or logistics experience is not required. You bring outstanding depth in agentic AI engineering, software engineering, and machine learning engineering, and you will work directly with AI, product, engineering, operations, data, and strategy partners on high-priority work.

Requirements

8+ years of hands-on software, machine learning, or applied AI engineering experience with Staff/Principal-level technical impact across teams—capable of independently taking a frontier-grade, full-stack agentic product from prototype to production (agent runtime, CLI/TUI, web frontend, backend services and APIs, and the underlying data layer) while remaining deeply involved in code, design, delivery, and production operations.
Co-equal depth in agentic AI engineering, software engineering, and machine learning engineering: ML and LLM fundamentals, experimental design, retrieval and ranking, model evaluation, production trade-offs, and sound judgment about when automation or an LLM is not appropriate.
Strong, hands-on experience with model post-training—reinforcement learning (RLHF/RLAIF), preference optimization, supervised fine-tuning, distillation, and reward and grader design—applied to improve reasoning, tool-use reliability, or domain-specialized model behavior.
Demonstrated experience shipping stateful, policy-controlled agent runtimes and tool-using multi-agent systems for autonomous long-horizon work: durable session lifecycle, context-window management, retrieval and RAG, structured outputs, guardrails, evaluation systems, secure delegation, safe tool use, interruption handling, and recovery—built on established orchestration frameworks or on custom infrastructure of your own design, including MCP and production tool and integration lifecycle management.
Broad applied ML depth across several areas such as reinforcement learning, multimodal modeling, retrieval and ranking, forecasting, optimization and operations research, experimentation, or model evaluation—together with production ML lifecycle and MLOps practices, and judgment about when a classical model beats an LLM.
Strong Python plus production backend capability in one or more of Go, Java, or TypeScript, with true full-stack capability (React/TypeScript frontend, APIs, and the data layer) that real operational users can trust.
Strong production experience with distributed systems, GCP or comparable cloud platforms, Kubernetes/Docker, CI/CD, data and streaming systems, telemetry and observability, SLOs and SLIs, reliability practices, secure enterprise integrations, and layered testing across unit, integration, workflow, and user-journey behavior—plus the ability to learn and operate within Walmart's internal technology ecosystem quickly.
Deep fluency with AI-native software-development workflows and coding agents such as Claude Code, Codex, Cursor, Copilot, or equivalent tools, applied with rigorous engineering judgment, code review, evaluation, and testing.
Demonstrated evidence of exceptional hands-on depth and impact beyond your own scope. We accept many forms of proof: shipped systems you can speak to in depth, an architecture and code walkthrough, open-source contributions where disclosure is permitted, technical demonstrations, publications, or patents. We recognize that much frontier work is confidential; a public portfolio is welcome but is not required. Be prepared to describe work that measurably improved the teams and products around you.
A highly motivated, proactive, ethical, and approachable builder with technical creativity and intellectual rigor, who challenges assumptions constructively, communicates clearly, and carries solutions from blank page to dependable production service.
Bachelor's degree in Computer Science, Computer Engineering, Computer Information Systems, Software Engineering, or a related area and 4 years of experience in software engineering or a related area; or 6 years of experience in software engineering or a related area.

Nice To Haves

Master's or Ph.D. in Computer Science, Computer Engineering, Machine Learning, Artificial Intelligence, or a closely related field.
Experience building developer-facing agentic platforms, SDKs, or coding and automation agents used by other engineers.
Graph and knowledge-representation depth is a strong plus. Knowledge and context graphs, ontology and schema design, entity resolution, multi-hop retrieval, graph databases and query languages, or graph ML and network analysis.
Experience with ML platform engineering, model serving, retrieval and ranking systems, experimentation platforms, and production LLM quality evaluation.
Experience building trusted associate-facing products, complex operational systems, high-volume data and streaming platforms, or reusable AI foundations.
Strong open-source contributions, technical talks, developer tools, or shipped agentic or ML products that demonstrate practical engineering depth.
Evidence of building innovative technologies or applications that improve complex operational workflows for frontline users, associates, customers, or other high-impact users.
Knowledge in implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly.
Knowledge of accessibility best practices and join us as we continue to create accessible products and services following Walmart’s accessibility standards and guidelines for supporting an inclusive culture.

Responsibilities

Build advanced agentic systems end to end: Design, build, test, launch, and operate production agentic applications and services with multi-step and long-horizon planning, tool use, retrieval, durable sessions and workflows, context management, multi-agent orchestration, human approval, and safe recovery when decisions or actions need intervention.
Build a policy-first agent runtime and control plane with deterministic allow, deny, and ask decisions; least-privilege tool and data access; scoped identity and authorization; auditable human approvals; bounded subagent delegation; and safe cancellation, user steering, and retry behavior for long-running work.
Build and own the full product path as a full-stack systems engineer: backend services and APIs, the data layer (database and schema design, data modeling, migrations, and streaming pipelines), telemetry, and accessible React/TypeScript experiences for associates and operators. Where the workflow demands it, design equally usable AI-native CLI/TUI or headless, structured-output interfaces that support automation, operations, and CI/CD-style integration.
Own the design and build of advanced multi-agent harnesses, runtimes, and orchestration capabilities using frameworks such as Pydantic AI, LangGraph, LangChain, AutoGen, or LlamaIndex, or purpose-built custom infrastructure. Building your own runtime where that is the right call is a strength, not a gap. Establish typed contracts, testability, scalability, operability, and developer ergonomics as non-negotiable platform properties.
Engineer explicit, inspectable agent execution. Design graph-based, state-machine-based, event-driven, planner/executor, or equivalent execution models as the problem warrants, with explicit state, typed dependencies and handoffs, conditional branching, fan-out and fan-in to subagents, retry and repair paths, loop detection, cost and time budgets, and hard termination conditions. Design external verification into the system—test runners, execution results, transaction outcomes, deterministic checks, and expert human review—because a model reviewing its own output is not verification.
Build governed knowledge and context infrastructure for durable agent memory. Choose and combine the right representations—knowledge and context graphs, vector retrieval, relational and temporal models, or hybrids—with an explicit schema and ontology treated as a product contract. Own entity resolution and conflict handling, construction and enrichment pipelines from structured and unstructured sources, provenance and temporal validity, schema validation and evolution, and multi-hop retrieval that answers questions flat retrieval cannot.
Build and operate agent skills, tool adapters, hooks and extension points, Model Context Protocol (MCP) clients and servers, structured outputs, function calling, enterprise APIs, identity, and secrets management. Own MCP lifecycle and reliability end to end: secure configuration and authentication, per-agent tool binding, schema compatibility, tool discovery, timeouts, health monitoring, retries, circuit breaking, quarantine, cleanup, failure isolation, and auditable operations.
Create governed extension ecosystems for agents, skills, commands, plugins, hooks, tool adapters, and reusable workflows. Define stable contracts, compatibility and versioning strategies, secure installation and update paths, isolation boundaries, rollback behavior, and observability so extensibility does not become an uncontrolled code-execution surface.
Treat agent quality as an engineering discipline: golden tasks, offline benchmarks, online experiments, adversarial and regression testing, failure analysis, measurable quality thresholds, and release gates.
Establish practical AgentOps / LLMOps practices for prompt and tool versioning, tracing, evaluation datasets, workflow reliability, cost and latency controls, incident learning, and continuous improvement of long-running autonomous systems. Instrument the system so engineers and operators can answer, with evidence, what the agent did, why it was permitted, which model, tool, and policy version was involved, what data and integrations were used, what it cost, where it failed, and how to reproduce or remediate the outcome safely.
Improve model reasoning and quality through hands-on post-training for the systems you own: reinforcement learning (RLHF/RLAIF), preference optimization, supervised fine-tuning, distillation, and reward and grader design to strengthen reasoning, tool-use reliability, and domain-specialized behavior across frontier models and smaller, domain-specialized models.
Build the model-improvement flywheel. Turn production interaction traces, tool-use trajectories, human feedback, and successful and failed reasoning paths—together with Walmart's proprietary enterprise and operational data, synthetic data, and curated evaluation sets—into governed training and evaluation datasets. Use them to post-train, distill, evaluate, and redeploy increasingly capable domain-specialized models back into the agentic systems, so the platform you build continuously improves the models that power it. Own the data governance, provenance, privacy, and access controls that make this safe at enterprise scale.
Design provider-aware, model-agnostic execution and routing layers that account for model capabilities—including multimodal inputs and outputs across text, images, and documents—context limits, structured outputs, streaming behavior, credentials, rate limits, transient failures, and explicit quality, latency, and cost trade-offs.
Use Google Cloud Platform (GCP) or comparable cloud platforms together with Walmart internal technologies to build systems that are secure, fault tolerant, cost-aware, observable, highly available, and low latency.
Build responsible enterprise-agent behavior through least-privilege access, explicit authorization boundaries, complete audit trails, data-protection controls, defenses against prompt injection and untrusted instructions, approval controls for consequential actions, and reversible recovery paths.
Design distributed data, streaming, and telemetry systems using technologies such as Kafka, Flink, Spark, OpenTelemetry, and Grafana; use operational signals to improve product quality, reliability, and user trust.
Set quality, performance, and release standards with layered unit, integration, API, workflow, and user-journey testing; adversarial safety testing; benchmark- and profile-driven performance work; CI/CD quality gates; reliable rollout and rollback practices; and incident-response readiness.
Deliver accessible associate-facing experiences that meet applicable Walmart accessibility standards, including WCAG 2.2 AA where applicable.
Mentor senior engineers through code, design, operational leadership, and clear technical judgment while staying immersed in implementation and customer and associate outcomes.
Raise the engineering bar through clear technical writing, rigorous code and architecture reviews, and reusable platform capabilities other teams can build on.
Make consequential technical trade-offs based on evidence, security, maintainability, measurable outcomes, and user needs—not novelty for novelty's sake.
Move quickly without cutting corners: create evidence-driven prototypes, validate them with real users and measurable evaluations, then harden successful ideas into production-grade systems.
Deliver strongly against your own responsibilities is the starting point for this position, not the measure of success. At the Principal level we expect you to make the teams and products around you measurably better. In practice, that means raising the technical performance of the engineers you work with, creating new capability rather than only consuming what already exists, taking ownership of difficult and high-consequence problems that others step around, and building foundations that keep paying off after the project that created them is finished.

Benefits

Performance-based bonus awards
Medical coverage
Vision coverage
Dental coverage
401(k)
Stock purchase
Company-paid life insurance
PTO (including sick leave)
Parental leave
Family care leave
Bereavement leave
Jury duty leave
Voting leave
Short-term disability
Long-term disability
Company discounts
Military Leave Pay
Adoption and surrogacy expense reimbursement
PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes.
Live Better U (Walmart-paid education benefit program for full-time and part-time associates in Walmart and Sam's Club facilities. Programs range from high school completion to bachelor's degrees, including English Language Learning and short-form certificates. Tuition, books, and fees are completely paid for by Walmart.)