Software Engineer, Machine Learning Infrastructure - Generative AI

DoorDash USA•Sunnyvale, CA

1d•$137,100 - $299,300•Remote

About The Position

DoorDash’s GenAI Platform team, within Machine Learning Platform, is responsible for building the shared infrastructure that enables teams across DoorDash, Wolt, and Deliveroo to safely deploy Generative AI-powered products, agents, automation, and personalization into production. The team's mission is to accelerate the business impact derived from GenAI. A key component of this mission is the evaluation platform, which serves as the unified backbone for measuring, tracing, and trusting the quality of LLM and agent systems company-wide. This platform supports trace/score ingestion, LLM-as-judge workflows, agent simulations, and LLM observability for millions of daily requests handled by the LLM Gateway. Additionally, the team manages core platform features such as the Agent Gateway, serving for open-weights models, batch inference, guardrails, and cost attribution. In this role, you will join a focused, high-impact team dedicated to building production infrastructure for Generative AI at DoorDash. Your primary focus will be on the evals and LLM observability platform, which empowers teams to evaluate, trace, and continuously enhance the quality of LLM and agent products. You will engage with evaluation frameworks and SDKs, OpenTelemetry-based trace/score ingestion, LLM-as-judge and offline/online evaluation pipelines, agent simulations, data pipelines, backend services, and observability tools. This position is well-suited for an engineer who thrives on developing robust measurement and quality primitives in a rapidly evolving technical landscape where product requirements, model capabilities, vendor ecosystems, and evaluation methodologies are constantly changing.

Requirements

B.S., M.S., or PhD. in Computer Science or equivalent
3+ years of industry experience in software engineering
Strong backend engineering fundamentals, especially in Python and distributed systems.
Experience building production services, APIs, data pipelines, or ML infrastructure at scale.
Experience operating systems in production, including observability, debugging, reliability, incident response, and performance/cost optimization.
Hands-on experience with evaluation, LLM observability, or measurement systems for ML/LLM products in production — eval pipelines, tracing/scoring, offline/online quality metrics, or experimentation.
Proficiency in using AI coding tools (e.g., Claude Code, Codex, Cursor) in the full software development lifecycle, including designing, generating code, testing, monitoring and releasing software

Nice To Haves

Depth in evaluation methodology — LLM-as-judge design and calibration, judge/eval drift detection, human-in-the-loop labeling, or eval harness design for agents and multi-step systems
Experience with LLM observability and tracing (e.g., OpenTelemetry, trace/score ingestion) and building instrumentation SDKs
Experience building and deploying AI agents or MCP servers in production, including agent evaluation or simulation
Experience with data pipelines, streaming ingestion, and analytical stores (e.g., SQL, columnar/OLAP) for high-volume telemetry
Experience with LLM gateways, model routing, vendor abstraction, or cost attribution
Experience building developer platforms, internal platforms, or self-serve infrastructure
Experience with Kubernetes, cloud infrastructure (AWS/GCP), or high-throughput batch systems
Experience with RAG, search, vector databases, or open-weights LLM inference and fine-tuning

Responsibilities

Build the infrastructure that helps DoorDash teams move GenAI ideas from prototype to production, increasing the velocity of business impact from AI across the company.
Work on our unified evals platform — evaluation SDKs, OpenTelemetry trace/score ingestion, LLM-as-judge, offline and online eval pipelines, and agent simulations — alongside the LLM Gateway, Agent Gateway, open-weights model serving, guardrails, and cost attribution.
Design scalable systems for evaluation workflows, trace/score ingestion, LLM observability, and agent simulation that power real customer and internal automation use cases.
Raise the quality bar for GenAI at DoorDash — giving product teams trustworthy, low-friction ways to measure model and agent quality, catch regressions, and compare across open-weight and closed-source models with observability and cost controls built in.
Build platforms that support rapid experimentation while meeting production standards for latency, scale, monitoring, SLOs, playbooks, and operational excellence.
Partner closely with ML engineers, product engineers, data scientists, and platform teams across DoorDash, Wolt, and Deliveroo to turn emerging GenAI capabilities into durable platform primitives.
Shape the future of DoorDash’s centralized GenAI platform — closing the loop from evaluation and agent observability to agent optimization, where eval signals and traces drive automated evaluation, agent simulation, and post-training techniques (e.g., reward modeling and RLHF/RLVR evaluation) — enabling the next generation of AI-powered products, agents, automation, and personalization.