Senior Software Engineer, ML Infrastructure

Gatik AI•Santa Clara, CA

About The Position

Gatik’s Scalable ML team builds the data generation, curation, orchestration, and distributed training infrastructure that enables machine learning developers to work efficiently with large volumes of real-world fleet data and simulated driving scenarios. The team works across ML Platform, Jobs Platform, Cloud Platform, and High Performance Computing, partnering closely with autonomy, simulation, safety, and data teams. The Senior or Staff engineer will design and build systems that transform raw driving data into high-quality training and evaluation sets, run training and evaluation reliably at scale, and provide metrics and tools that support confident release decisions for driverless operations. At the Staff level, the role also sets multi-quarter technical direction and serves as a technical bridge across teams. The work includes building reproducible data pipelines; curating balanced datasets and assessing their coverage and quality; scaling distributed training, inference, and evaluation on multi-tenant GPU clusters; establishing MLOps practices; defining statistically rigorous model and release metrics; designing and communicating experiments; building engineer-facing dashboards and services; shaping shared compute infrastructure; and improving engineering quality and reliability. Staff-level responsibilities include driving technical strategy, representing Scalable ML in architecture discussions, and mentoring engineers.

Requirements

Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, Robotics, or a related field and 5+ years of relevant industry experience, or a Master’s degree and 3+ years.
Strong Python software engineering skills, including designing modular, maintainable, high-performance, well-tested systems in a shared codebase.
Experience building end-to-end ML systems, from data pipelines and dataset curation through training, evaluation, and production, using tools such as SQL, pandas, PyTorch, or scikit-learn.
Hands-on production experience with Ray (Ray Core, Ray Data, Ray Train, and/or Ray Tune), including running and debugging Ray workloads at scale.
Deep experience with PyTorch and PyTorch Lightning for large-scale training, including distributed strategies such as DDP or FSDP, mixed precision, and checkpointing.
Experience designing and operating scalable ML infrastructure, such as distributed training and evaluation pipelines, experiment tracking, model serving, or large-scale distributed jobs.
Familiarity with shared-compute environments, including GPU scheduling and quota, job orchestration, and multi-tenant clusters.
Solid grounding in statistics and experimental design, including selecting evaluation strategies, metrics, and thresholds for real-world decisions.
Clear communication skills and ability to work effectively across software, ML, and autonomy teams.

Nice To Haves

Staff candidates typically bring 8+ years of experience, including time setting technical direction beyond a single team.

Responsibilities

Design and maintain reproducible pipelines to discover, label, and organize driving scenarios from fleet and simulation data, including versioning, provenance, train/validation/holdout splits, and data contracts.
Build balanced training and evaluation datasets covering truck scenarios across routes, weather, and traffic conditions; analyze coverage, composition, and label quality.
Build distributed training, inference, and evaluation pipelines with robust data loading, checkpointing, and experiment tracking for multi-tenant GPU clusters.
Implement model artifact management, experiment tracking, automated model testing, and deployment practices for adoption across Gatik.
Define, implement, and validate model and release metrics, and characterize behavior across software releases and scenario types.
Establish baselines, ablations, and holdout comparisons; quantify uncertainty and communicate experiment results to support autonomy-team decisions.
Develop dashboards and services for comparing software versions, exploring scenario-level results, visualizing model outputs, and collecting reviewer feedback.
Partner with ML Platform, Jobs Platform, Cloud Platform, and HPC engineers on GPU scheduling and quota, job orchestration, storage, and compute cost.
Design clean interfaces between data, modeling, and product layers; promote testing, type checking, code review, and fast CI.
Lead incident root-cause analysis and prevention, including incidents that cross team boundaries.
At Staff level, define and drive multi-quarter technical strategy for data and training infrastructure, represent Scalable ML in architecture discussions, and mentor engineers.