Member of Technical Staff - ML Ops

Vinci • Palo Alto, CA

11d • $185,000 - $245,000

About The Position

Vinci is building a foundation model that learns physics from simulations, with the goal of making physical reasoning more accessible to engineers. Its simulation data corpus operates at petabyte scale and is growing rapidly. The Member of Technical Staff - ML Ops will build and operate the systems that turn simulation-generation recipes into versioned, validated datasets that training runs can consume, while expanding the job scheduling and execution platform to support millions of simulations, ML training runs, and future workloads. The role is hands-on in a production codebase and involves writing, testing, and shipping code.

Requirements

• At least 5 years building production data or ML pipelines, including at least 2 years owning a pipeline running on GPUs at meaningful scale.
• Experience running large batch or simulation workloads involving thousands of long-running jobs, partial failures, retries, and preemption, while preventing work from being lost or duplicated.
• Strong Python and software engineering fundamentals, including clean abstractions, typed code, and a trusted test suite.
• Comfort with relational and NoSQL databases at scale, including schema design, live-data migrations, bulk writes, and query performance on tables with hundreds of millions of rows.
• Experience with workflow orchestration or distributed execution frameworks such as Temporal, Ray, Airflow, Argo, Kubernetes Jobs, or Slurm.
• Working knowledge of cloud object storage and file formats used to feed training at high throughput.
• A focus on enabling others to run experiments, generate data, and catch failures early.

Nice To Haves

• PyTorch distributed training, including DDP, FSDP, torchrun, or Ray Train; checkpointing; and experiment tracking.
• MPI, CUDA, or HIP experience, or experience operating numerical solvers in production.
• Infrastructure as code with Terraform or Ansible, and private networking for multi-environment deployments.
• Scientific or simulation data experience, such as CFD, FEA, EDA, CAD, or other large numerical or geometric datasets.
• Experience designing parameter-sampling or curriculum strategies for training-data coverage.
• Prometheus, OpenTelemetry, or similar observability-stack experience.

Responsibilities

• Scale the data-generation pipeline from thousands of simulations to millions.
• Own dataset lineage and reproducibility, recording how each simulation was produced and how it can be augmented for future generation jobs.
• Build data-quality measures into the pipeline, including acceptance criteria, bad-sample flagging, target-space coverage and distribution checks, and per-field statistics.
• Extend and operate the GPU job platform, which uses Temporal as a durable control plane and Ray as the execution plane, including priority, admission, cancellation, and isolated per-job environments.
• Support generation, training, and validation workloads competing for the same compute fleet.
• Own training-job launch using Ray Train or torchrun, checkpoint management, the model registry, and paths for trained checkpoints to feed future generation campaigns or production inference pipelines.
• Work with the infrastructure team to ensure compute and data resources meet workload needs.
• Make costs visible by physics type and configuration, identify GPU-time inefficiencies, and inform decisions about which data to generate.
• Maintain a healthy codebase by writing tests, keeping CI fast, creating safe schema migrations for populated databases, and documenting systems for operation by other engineers.

Benefits

• Work on production programs at leading semiconductor companies, with work that contributes to real hardware.
• Work on a challenging problem at the intersection of AI, first-principles physics, and enterprise engineering.
• Work alongside experienced engineers and researchers.
• Fast shipping, direct ownership, and input that shapes the product and company.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed