Senior Staff Machine Learning Performance Engineer

XPENG • Santa Clara, CA

16h • $244,140 - $413,160

About The Position

XPENG is building the next generation of end-to-end autonomous driving on physical AI foundation models, including vision-language-action policies and world models. The same training stack supports the company's robotics program. As a Senior Staff Machine Learning Performance Engineer, you will own training speed and performance across large-scale workloads, work with modeling teams on challenging bottlenecks, set technical direction, and turn successful optimizations into reusable defaults. You will also develop agentic workflows to automatically identify and resolve performance opportunities.

Requirements

• Master's or PhD in computer science, computer engineering, or electrical engineering, or equivalent practical experience.
• At least 8 years of industry experience, the majority in performance engineering.
• Hands-on experience optimizing GPU training or inference workloads in production at multi-node scale.
• Demonstrated ability to diagnose performance problems in production workloads and deliver fixes, often in systems owned by other teams, with gains measured and verified in production.
• In-depth understanding of computer architecture and high-performance computing.
• Strong C++ and Python skills, with a solid command of data structures and algorithms.

Nice To Haves

• Hands-on experience writing and tuning CUDA or Triton kernels.
• Fluency in PyTorch and concrete insight into performance factors such as graph breaks, host-side overhead, synchronization points, and memory layout.
• Experience training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
• Experience building profiling, tracing, or benchmarking infrastructure used by other teams, such as CUPTI, Perfetto, or equivalent.
• Experience using LLM agents to automate parts of an engineering workflow.
• Experience in autonomous driving, robotics, or another domain with hard latency, power, and memory constraints.
• Publications or open-source contributions in ML systems or performance.

Responsibilities

• Own step time, scaling efficiency, and long-run stability for foundation-model training across thousands of accelerators, covering the data path, kernels, parallelism, and communication.
• Work with modeling, infrastructure, simulation, data-mining, and robotics teams to address difficult performance bottlenecks and set performance direction.
• Turn individual performance improvements into modular components that become defaults across models and workloads.
• Extend performance tooling and metric coverage across the stack while keeping instrumentation overhead low enough for fleet-wide production use.
• Build agents that can reproduce jobs on demand, verify numerical parity, and distinguish genuine regressions from run-to-run noise.

Benefits

• Fun, supportive, and engaging environment.
• Infrastructure and computational resources to support your work.
• Opportunity to work on cutting-edge technologies with top talent in the field.
• Opportunity to make a significant impact on the transportation revolution by advancing autonomous driving.
• Competitive compensation package.
• Snacks, lunches, dinners, and fun activities.
• Bonus, equity, and benefits.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior