Machine Learning Infrastructure Engineer

True Talent Solutions Partners•San Francisco, CA

2d•Onsite

About The Position

Join an AI team developing large foundation models for physical systems, with weather as a core application. The role builds infrastructure for scalable model training and inference to advance AI for science and physical AI. The Machine Learning Infrastructure Engineer designs, deploys, and maintains distributed machine learning training and inference clusters; builds end-to-end pipelines for large datasets and model training across the machine learning lifecycle; tests training approaches, including parallelization methods and numerical precision trade-offs; profiles and debugs low-level GPU operations to improve performance; and follows relevant research to apply useful ideas to infrastructure and training workflows.

Requirements

Two to ten years of experience building large-scale machine learning infrastructure for foundation models, including compute clusters, data infrastructure, inference, and distributed training.
Experience optimizing large-scale training, data, or inference workloads.
Expertise with distributed training frameworks such as FSDP or DeepSpeed.
Proficiency with Python and C++.
Experience with GPU performance optimization, CUDA, or JAX.
Familiarity with cloud platforms such as GCP, AWS, or Azure, plus Docker, Kubernetes, and scalable model serving or task management systems.
Background in physical AI or AI for science, such as robotics, biology, or other science-focused applications.
Experience across the machine learning lifecycle, including analyzing training experiments and ablation studies, and working in fast-paced environments.

Responsibilities

Design, deploy, and maintain distributed machine learning training and inference clusters.
Build scalable end-to-end pipelines for large datasets and model training across the machine learning lifecycle.
Test training approaches, including parallelization methods and numerical precision trade-offs across model scales.
Profile and debug low-level GPU operations to improve performance.
Follow relevant research and apply useful ideas to infrastructure and training workflows.