Senior Staff Machine Learning Performance Engineer

Xpeng•Santa Clara, CA

About The Position

XPENG is hiring a Senior Staff Machine Learning Performance Engineer to improve the speed, scaling efficiency, and stability of training next-generation end-to-end autonomous-driving foundation models, including vision-language-action policies and world models. The training stack also supports the company's robotics program. Training runs can span thousands of accelerators and multiple days, so the role owns performance across the stack, from data paths and kernels to parallelism and communication. The engineer will work with modeling and other technical teams to set performance direction, diagnose bottlenecks in production workloads, and turn successful optimizations into reusable defaults. The role also involves expanding performance observability while keeping instrumentation overhead low, and building agentic workflows that reproduce jobs, verify numerical parity, and distinguish regressions from run-to-run noise. XPENG describes its environment as fun, supportive, and engaging, with infrastructure and computational resources, opportunities to work with leading talent on cutting-edge technologies, and the chance to contribute to autonomous driving and transportation. The compensation package is competitive. The base salary range for this full-time position is $244,140–$413,160, plus bonus, equity, and benefits. Pay within the range depends on work location and factors such as skills, experience, education, and training. XPENG is an equal opportunity employer.

Requirements

Master's or PhD in computer science, computer engineering, or electrical engineering, or equivalent practical experience.
At least 8 years of industry experience, most of it in performance engineering.
Hands-on experience optimizing GPU training or inference workloads in production at multi-node scale.
Demonstrated ability to diagnose and fix performance problems in real production workloads, including systems owned by other teams, and measure and verify gains in production.
In-depth understanding of computer architecture and high-performance computing.
Strong C++ and Python skills, with a solid command of data structures and algorithms.

Nice To Haves

Hands-on experience writing and tuning CUDA or Triton kernels.
Fluency in PyTorch and concrete understanding of factors that affect performance, including graph breaks, host-side overhead, synchronization points, and memory layout.
Experience training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
Experience building profiling, tracing, or benchmarking infrastructure used by other teams, such as CUPTI, Perfetto, or equivalent.
Experience using LLM agents to automate parts of an engineering workflow.
Experience in autonomous driving, robotics, or another domain with hard latency, power, and memory constraints.
Publications or open-source contributions in ML systems or performance.

Responsibilities

Own step time, scaling efficiency, and long-run stability for foundation-model training across thousands of accelerators, covering the data path, kernels, parallelism, and communication.
Work with modeling, infrastructure, simulation, data-mining, and robotics teams to address difficult performance bottlenecks and set technical performance direction.
Turn individual performance improvements into modular components that become defaults across models and workloads.
Extend performance tooling and metric coverage across the stack while keeping instrumentation overhead low enough for fleet-wide production use.
Build agents that can reproduce jobs on demand, verify numerical parity, and distinguish actual regressions from run-to-run noise.