Principal ML Engineer

Electronic Arts•Vancouver, BC

9h•CA$204,600 - CA$287,500•Hybrid

About The Position

Electronic Arts is hiring a Principal ML Engineer, Training Pipelines, to own systems that prepare and deliver multimodal game data for large-scale model training. The role sets architecture for training input pipelines and improves data throughput, accelerator utilization, reproducibility, and failure recovery. It begins as a hands-on individual contributor and technical leadership position, with the potential to take on people-management responsibilities as the team grows. The engineer partners with researchers and central data and compute teams to translate evolving model requirements into reusable training capabilities. The position focuses on shared, scalable training input systems, from certified game capture through delivery to accelerators. It encompasses data schemas and lifecycle standards, deterministic and recoverable data loading, reproducible multimodal sampling, distributed performance and reliability measurement, diagnosis of cross-layer bottlenecks, and integration with training frameworks, optimized runtimes, kernels, and accelerator hardware. The successful candidate will lead architecture reviews, mentor engineers, and coordinate priorities with research leads, while researchers retain ownership of model research.

Requirements

20+ years of development experience, including ownership of production systems for distributed-training input, with measurable improvements in throughput, accelerator utilization, step time, reliability, recovery time, or cost.
Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for single jobs spanning thousands of accelerators.
Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and optimization on one or more accelerator platforms such as GPUs, TPUs, or AWS Trainium.
Pre-training experience and understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency.
Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.
Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, and elastic restart.
Experience with schemas, dataset versioning, lineage, quality, and reproducible training/evaluation.
Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults.

Responsibilities

Architect and build shared training input pipelines from certified game capture to accelerators, including storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across multiple training frameworks.
Define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, acceptance criteria, and reproducible splits.
Build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and corrupt data, including elastic restarts and changes to cluster size or parallelism.
Design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, titles, and capture conditions, accounting for worker/rank locality and distributed parallelism.
Benchmark and enforce throughput, data-wait time, step time, accelerator utilization and stalls; model MFU where applicable, per-rank skew, scaling efficiency, restart time, reliability, and cost targets through thousands of accelerators.
Diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators, including stragglers, faulty hardware, corrupt samples, and imbalanced batches.
Integrate compilers, runtimes, optimized kernels, low-precision training, and hardware-accelerated media processing; maintain efficient data delivery and recovery under FSDP/HSDP, tensor, context, and pipeline parallelism across heterogeneous accelerator environments.
Lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads.
Translate model and data-access requirements into scalable systems that build on shared infrastructure services, while researchers retain model-research ownership.

Benefits

Paid time off (3 weeks per year to start) for US employees; vacation (3 weeks per year to start) for Canada employees.
80 hours per year of sick time in the US; 10 days per year of sick time in Canada.
16 paid company holidays per year in the US.
10 weeks paid time off to bond with a baby in the US.
Paid top-up to EI/QPIP benefits up to 100% of base salary when welcoming a new child in Canada: 12 weeks for maternity and 4 weeks for parental/adoption leave.
Medical, dental, and vision insurance in the US; extended health, dental, and vision coverage in Canada.
Life insurance and disability insurance.
401(k) for regular full-time US employees; retirement plan for regular full-time Canadian employees.
Certain roles may be eligible for bonus and other incentive programs.
Benefits may include mental well-being support, retirement savings, paid time off, family leaves, complimentary games, and healthcare coverage, tailored to local needs.