Senior / Principal Infrastructure Engineer - ML Platform

Roblox•San Mateo, CA

2d•$278,530 - $345,040•Hybrid

About The Position

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. ML Platform @ Roblox today supports hundreds of ML use cases and billions of inferences per day across Discovery, Safety, Engine, and much more. As an Infrastructure Engineer on the ML Platform team, you will design, scale, and maintain the foundational infrastructure powering our entire machine learning ecosystem. We are looking for accomplished engineers to spearhead the development of our next-generation ML tooling and platform capabilities.

Requirements

6+ years of professional experience and a tool chest of system design experience upon which to draw to build scalable, reliable platforms.
Deep experience with Kubernetes (K8s) and cluster management at scale — e.g., managing 100s–1000s of nodes, serving 100k+ QPS, and ideally having experience writing custom Kubernetes controllers.
Strong proficiency in Infrastructure as Code (IaC), specifically using Terraform to bootstrap, manage, and automate cloud infrastructure across AWS, GCP, or similar environments.
Bachelor's degree in Computer Science, Computer Engineering, Data Science, or a similar technical field or equivalent practical experience.

Nice To Haves

Proficient in DevOps tooling such as Docker, Kubernetes, CI/CD systems, and bootstrapping cloud infrastructure (AWS, GCP, etc.)
Experienced with the end-to-end ML model lifecycle such as model serving, training, model CI/CD, and GPU resources management, and have built ML platform features that are delightful to use.
An automation advocate: you're passionate about infrastructure-as-code and automating painful manual processes.
A reliability nut: you love digging into tricky postmortems and identifying weaknesses in complicated systems.
Passionate about supporting internal partners (data scientists and ML Engineers) to meet and understand their needs.

Responsibilities

Bootstrap and maintain Kubernetes and Cloud infrastructure for ML Platform components--Serving Layer, Metadata Store, Model Registry, and Pipeline Orchestrator.
Set technical strategy and oversee development of high scale and reliable infrastructure systems.
Propose and implement new platform tooling to improve time to production for MLEs and Data Scientists across the full ML lifecycle.
Work on infrastructure projects such as GPU fleet management, hybrid-cloud orchestration, and writing custom Kubernetes controllers and resources.
Stay abreast of industry trends in machine learning and infrastructure to ensure the adoption of leading-edge technologies and practices.
Partner across organizations to build tooling, interfaces, and visualizations that make the ML@Roblox a delight to use.