ML Research Engineer, Training

Weave Robotics•San Francisco, CA

About The Position

Weave is hiring an ML Research Engineer to build the training stack that turns large-scale real-world robot data into model capability. The company operates robots in homes and businesses, producing terabyte-scale fleet data. This role owns the training pipeline from raw fleet uploads through data preparation and sampling to GPU training, helping research ideas become models in training within days and deployed within weeks. The position focuses on making training infrastructure reliable, performant, scalable, and productive for the research team. It includes distributed training, data systems, experiment orchestration, monitoring and debugging, and translating research prototypes into infrastructure used across the team.

Requirements

  • Deep experience with PyTorch or JAX, including multi-node distributed training such as FSDP, DDP, or equivalent on real workloads.
  • Experience profiling and optimizing GPU utilization, data pipelines, I/O bottlenecks, memory usage, and distributed training performance.
  • Experience with CUDA-level profiling tools such as Nsight Systems and with NCCL tuning.
  • Sound judgment about training runs, including interpreting loss curves, distinguishing instability from data bugs, and deciding when to stop a run.

Nice To Haves

  • Robot learning experience, including training policies such as VLAs, world models, or reinforcement learning, and distinguishing data problems from model problems.
  • Cluster and cloud infrastructure experience with Kubernetes, SLURM, GCP, or AWS.
  • Large-scale post-training experience, including SFT, reward modeling, or RL fine-tuning.
  • On-robot inference optimization experience, including TensorRT, quantization, or distillation.
  • CUDA or Triton kernel development experience.
  • Experience with video-heavy datasets, including transcoding, chunking, and storage/compute tradeoffs for training on video at scale.

Responsibilities

  • Build the end-to-end training stack, including distributed training, data loading, checkpointing, run orchestration, and experiment tracking.
  • Develop high-throughput ingestion, transformation, and storage systems for terabytes of multimodal robot data, including video, proprioception, and sensor streams.
  • Improve research productivity by making experiments reproducible, scalable, and easy to launch, monitor, retry, recover, and debug.
  • Drive data sampling and curation decisions that affect model behavior.
  • Profile and fix throughput bottlenecks, investigate loss spikes and silent data bugs, and make results reproducible and trustworthy from data loading to GPU kernels.
  • Turn research prototypes into infrastructure used by the whole team for training.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service