Machine Learning Performance Engineer - Offboard Training & Inference

Applied Intuition•Sunnyvale, CA

8d•Onsite

About The Position

Applied Intuition is seeking a performance engineer specializing in making large-scale machine learning workloads fast and cost-efficient in the datacenter. This role focuses on distributed training runs spanning many nodes and high-throughput batch inference for processing large datasets. The primary optimization targets are throughput, cluster goodput, and cost per unit of data processed, aiming to reduce inefficiencies in GPU-hours and processing time. The engineer will be responsible for profiling across the stack, identifying performance bottlenecks, and implementing solutions to bridge the gap between theoretical accelerator capabilities and actual workload performance. This involves working at the intersection of accelerators, ML frameworks, and large-scale data infrastructure, collaborating with various teams to improve training time-to-result and offline processing costs. Engineers at Applied Intuition are encouraged to take ownership of technical and product decisions, interact with users, and contribute to a dynamic team culture.

Requirements

Hands-on ML performance engineering experience: profiling, roofline analysis, throughput optimization, and root-cause investigation in production systems
Experience with distributed multi-node training at scale (FSDP, DeepSpeed, Megatron, NCCL, or equivalent), including diagnosing scaling inefficiency as node count grows
Deep familiarity with GPU or accelerator performance concepts - memory bandwidth, kernel launch overhead, occupancy, quantization, collective communication
Experience with high-throughput or batch inference systems (NVIDIA Triton Inference Server, TensorRT, ONNX Runtime, Ray, or similar)
Fluency in Python and proficiency in C++ or another systems language
Excellent debugging, analytical, and problem-solving skills
A deep understanding of machine learning foundations, and the ability to develop technical solutions for problems with no established playbook

Nice To Haves

GPU kernel development experience: CUDA, Triton, CUTLASS, or hand-tuned attention implementations
Experience with profiling toolchains such as Nsight Systems/Compute, PyTorch Profiler, or perf
Experience with GPU scheduling and orchestration on Kubernetes, Slurm, or Ray, including multi-tenant cluster utilization
Experience with fault tolerance and elastic training for long-running jobs - checkpointing strategy, straggler mitigation, preemption recovery
Familiarity with autonomy or robotics data (ROS, OpenCV, multi-sensor log formats)

Responsibilities

Profile and optimize distributed training end to end - data loading and preprocessing, augmentation, kernel execution, gradient communication, and checkpointing
Optimize large-scale offline and batch inference over petabyte-scale sensor logs: batching and scheduling strategies, quantization and low-precision execution, graph optimization, and accelerator saturation across long-running sweeps
Establish roofline and performance models for our workloads, quantify the gap between achieved and theoretical performance, and stack-rank optimization opportunities by impact and effort
Improve multi-node scaling efficiency: sharding and parallelism strategies, collective communication, interconnect utilization, and memory-bandwidth and kernel-fusion bottlenecks
Drive cluster goodput - reduce GPU idle time from input pipeline stalls, storage and network I/O, scheduling gaps, stragglers, and failure recovery on long-running jobs
Build the benchmarking, observability, and regression-detection tooling that keeps performance from silently degrading as models and code evolve
Collaborate with engineers across functions to solve complex data and compute problems at scale
Contribute to a team culture that values effective collaboration, technical excellence, and innovation