Principal Model Optimization Engineer

Roblox•San Mateo, CA

4h•$295,250 - $345,040•Onsite

About The Position

Roblox is hiring a Principal Engineer on its ML Platform team, which supports hundreds of machine learning use cases and billions of inferences per day across Discovery, Safety, Engine, and other areas. The team builds the platform used to take ML ideas into production, including data and training pipelines, SDKs and tools, GPU infrastructure, and job scheduling. In this role, you will set technical direction for ML Platform systems, identify sources of friction for ML engineers and wasted compute, and design durable improvements. You will measure results and iterate to improve engineering speed, job reliability, and GPU utilization. You will work across engineering teams and mentor senior engineers. For office-based roles, employees work onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday, unless otherwise noted. The posting identifies the San Mateo, California base-pay range as $295,250–$345,040 annually. The actual base pay may vary based on job-related factors. Full-time employees are eligible for equity compensation and benefits.

Requirements

At least 6 years of professional experience and system design experience building performant systems.
A track record of building ML infrastructure end to end, such as training pipelines, workflow orchestration, data or feature pipelines, experiment tracking, model registries, or model serving.
Experience designing widely adopted SDKs, APIs, or developer tools, and understanding what makes a platform easy to use.
Hands-on experience running GPU workloads at scale on Kubernetes, including scheduling, quotas, multi-cluster dispatch, autoscaling, and debugging GPU and node failures such as Xid errors and NCCL hangs.
Knowledge of training and inference performance, including distributed training, mixed precision, batching, and quantization.
A habit of setting clear success metrics, shipping in steps, and iterating based on results.
Experience leading cross-team technical work without direct authority and writing clearly for engineers and leaders.
A Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.

Responsibilities

Design platform capabilities that can be adopted across many teams and use cases.
Build GPU and compute management for a scarce, mixed fleet, including quotas, priority tiers, preemption, cross-cluster scheduling, and capacity planning with Infrastructure.
Define, instrument, and improve platform success metrics, such as time from code to first training run, job success rate, GPU utilization, and cost per training run or inference.
Investigate slow or failing jobs across pipelines, schedulers, GPU profiles, and hardware errors, and implement durable fixes.
Work with ML engineers and data scientists to understand their workflows and make the platform’s paved path easy to use.
Partner with Infrastructure, Data Platform, and Security on shared foundations, including storage, networking, data access, and compliance.
Raise the technical bar through design reviews, written proposals, and code, and mentor senior engineers.