AI/ML Training Performance Engineer

Blackrock Neurotech•Salt Lake City, UT

50d•Onsite

About The Position

The AI/ML Training Performance Engineer owns the efficiency and scalability of model training on GPU and cloud infrastructure, improving the speed and capability of experiments as model training grows in complexity and compute requirements. This is a hands-on individual contributor role on a small research team, working across the training stack from Python and model execution through GPU kernels, distributed communication, and runtime environments. The engineer partners with model researchers, data engineers, and infrastructure and IT teams to identify and implement performance improvements while preserving numerical correctness and scientific intent. The role has significant ownership over how training performance is measured, optimized, and scaled, including establishing baselines, tooling, and technical approaches to support the growth of AI/ML work. It is based on-site at Blackrock Neurotech’s headquarters in Salt Lake City, Utah; occasional travel may be required. The team values ownership, accountability, sustained high-quality work, rigor, thoughtful decision-making, and deliberate collaboration. The role is high-ownership but is not described as always-on.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience, with 3+ years of relevant industry experience; alternatively, a master’s degree with 1+ years of relevant industry experience, or a PhD in a related field.
Demonstrated experience improving the performance of substantial deep learning training workloads, with measured gains in speed, memory efficiency, or compute cost.
Measurement-driven performance optimization using profiling and benchmarking to validate meaningful end-to-end improvements.
Exceptional programming ability in Python and C++ or a comparable systems language, with strong debugging, testing, and performance analysis practices.
Deep understanding of GPU execution, including memory hierarchies, memory coalescing, thread blocks, warps or wavefronts, occupancy, synchronization, and bandwidth limits.
Hands-on experience developing and profiling GPU kernels with CUDA or HIP/ROCm, and ability to diagnose correctness and performance at the hardware level.
Deep knowledge of PyTorch or an equivalent framework, including automatic differentiation, computation graphs, tensor storage, compilation, and mixed-precision training.
Strong understanding of deep learning architectures and the computations that drive training performance.
Experience with distributed training, collective communication, sharding, and the interaction between model partitioning and GPU interconnects.
Strong understanding of numerical stability and ability to validate gradients, convergence, and model quality after performance changes.
Experience configuring and diagnosing Linux-based GPU environments, containers, cloud compute, and high-throughput storage or networking.
Ability to collaborate closely with researchers and infrastructure teams and make clear tradeoffs between implementation effort, performance, reliability, and scientific value.

Nice To Haves

Experience with Triton.
Experience with compiler optimization.
Experience with advanced GPU profiling tools.
Experience with multiple accelerator generations.
Experience with long-sequence or multimodal models.
Experience with neural time series.
Experience with large-scale model training.

Responsibilities

Own training performance across single-GPU, multi-GPU, and multi-node workloads, establishing reproducible baselines for throughput, memory use, utilization, time to target quality, and cost.
Profile the full training path to identify compute, memory, communication, CPU, and I/O bottlenecks, and prioritize changes with measurable end-to-end impact.
Optimize tensor layouts, precision, memory allocation, activation checkpointing, operator fusion, and execution graphs to fit larger or longer-context models within available resources.
Write, tune, and validate custom GPU kernels using CUDA, Triton, HIP/ROCm, or appropriate platform tools when existing implementations limit performance.
Improve Python training scripts, framework and compiler settings, batching, gradient accumulation, and optimizer execution while preserving intended training behavior.
Design and tune distributed training strategies, including data, tensor, pipeline, or sharded parallelism, based on model structure, memory limits, and interconnect topology.
Partner with model researchers on hardware-aware architecture and hyperparameter changes, measuring effects on convergence, model quality, and compute requirements.
Coordinate with the neural data infrastructure engineer on prefetching, pinned memory, host-to-device transfer, and I/O overlap so data delivery keeps pace with training.
Work with infrastructure and IT on GPU selection, cloud instance configurations, networking, drivers, containers, scheduling, and capacity planning as training needs and compute capacity scale.
Build robust checkpoint, restart, and recovery workflows and performance regression checks to keep long-running experiments reproducible and productive.
Communicate benchmark evidence, numerical tradeoffs, scaling limits, and resource recommendations to researchers and organizational stakeholders.