Engineer Jobs

About The Position

Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
• 12+ years of relevant software engineering experience, including hands-on optimization of production GPU training or inference workloads.
• Demonstrated performance work across multiple layers of the stack, with a clear account of the baseline, evidence, alternative explanations, and validated outcome.
• Experience diagnosing performance across many jobs, machines, or tenants, rather than tuning only a single workload.
• Familiarity with framework, runtime, and compiler interfaces sufficient to determine where signals should be measured, even when others implement the instrumentation.
• Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and measurement effects, and identify when evidence is insufficient.
• Strong programming and debugging skills, demonstrated through maintainable analysis code, tests, or repeatable investigation procedures, including when using automated development tools.
• Ability to communicate findings, trade-offs, and remaining uncertainty clearly to workload owners.
• Proficiency with Python and CUDA.

Nice To Haves

• Experience improving both training and inference operations.
• Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang, or Dynamo.
• Experience across application, system software, driver, or hardware roles, and the full-stack perspective that comes with it.
• Statistical judgment applied to noisy, multi-node, fleet-scale performance data; ideally, experience with counterfactual analysis and workload segmentation for regression diagnosis.
• Experience building debugging or performance-analysis tools used by other engineers, with evidence that the tools made investigations more effective.

Responsibilities

• Analyze distributed training and inference performance across models and frameworks, data loading and checkpointing, compilers and kernels, CUDA and NCCL, hosts, and networks; identify bottlenecks, resolve them or recommend solutions, and provide supporting evidence.
• Define performance metrics tracked across GPU fleets, break metrics down into factors engineers can use for debugging, and determine which signals the platform should collect first.
• Lead regression analysis by segmenting workloads across machines, versions, and configurations to isolate causes, and develop reusable analysis methods.
• Evaluate automated diagnoses for accuracy, identify misleading results and missed problems, and create labeled cases to validate the analysis.
• Work with build partners on their workloads, drive findings through to improvements, and provide product feedback to the roadmap.

Benefits

• Equity
• Benefits