ML Infra Engineer, Compute

Physical Intelligence•San Francisco, CA

About The Position

Physical Intelligence is developing general-purpose AI for the physical world. The ML Infrastructure team supports its modeling work by building systems that make large-scale training reliable, reproducible, and fast. This role owns the compute scheduling and placement problem end-to-end, including scheduling systems, placement logic, cluster management, and operational tooling. The goal is to route training jobs automatically to appropriate clusters across a heterogeneous fleet of GPU and TPU hardware, cloud providers, and cluster topologies, so researchers can focus on their work without selecting clusters or configuring jobs for specific resources. This is a systems engineering role focused on intelligent resource allocation, utilization, fault tolerance, and seamless large-scale distributed training—not simply setting up cloud clusters. The position works closely with ML infrastructure, data platform, and research teams, and also extends scheduling and orchestration to inference workloads and model deployment to robot edge devices.

Requirements

  • Strong software engineering fundamentals.
  • Experience building or operating job scheduling or resource management systems at scale.
  • Experience with large-scale compute clusters using GPUs and/or TPUs.
  • Familiarity with schedulers and orchestration systems such as SLURM, Kubernetes, GKE, K3S, or internal equivalents.
  • Ability to reason about resource allocation, bin-packing, priority scheduling, and multi-tenancy.
  • Understanding of ML training workloads, including their long-running, multi-node, straggler-sensitive, and topology-dependent characteristics.
  • A willingness to own systems end-to-end, from design through operation.
  • Enjoyment of working closely with researchers and unblocking fast-moving projects.

Nice To Haves

  • Experience building multi-cluster or federated scheduling systems.
  • Experience with TPU infrastructure, such as GCP TPU slices, Multislice, or GKE.
  • Background in cluster resource managers such as Borg, YARN, Mesos, or custom schedulers.
  • Linux systems engineering, networking, and infrastructure-as-code experience.
  • Experience with NCCL, collective communication, and topology-aware placement.
  • Experience with capacity planning and cloud cost optimization at scale.
  • Familiarity with JAX, PyTorch, or similar ML frameworks at the runtime or systems level.

Responsibilities

  • Design and build multi-tenant scheduling systems that place training jobs on suitable clusters according to hardware requirements, topology, availability, cost, and priority.
  • Support fair resource sharing across teams and projects through quota management, priority tiers, and preemption policies.
  • Abstract cluster differences so researchers can submit jobs without choosing a destination cluster.
  • Build a control plane to manage job lifecycles across diverse GPU and TPU clusters, hardware generations, and on-premises and cloud environments.
  • Enable seamless job migration, failover, and rescheduling across clusters.
  • Monitor and optimize GPU and TPU utilization across the compute fleet.
  • Implement priority, preemption, queueing, and fairness policies that balance research velocity and cost efficiency.
  • Implement fault detection, automatic recovery, and resilience for long-running, multi-node training jobs.
  • Manage health checking, node management, and scaling to thousands of accelerators.
  • Extend scheduling and orchestration to inference workloads, including deploying models to edge devices on physical robots.
  • Build dashboards, alerting, SLOs, and debugging tools to help researchers understand job status and maintain scheduling quality and cluster reliability.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service