Software Engineer, ML Infrastructure

Arc Institute•Palo Alto, CA

2d•$191,000 - $237,000•Hybrid

About The Position

You'll join Arc's Infrastructure Engineering team and own the data and evaluation of ML infrastructure for Arc's virtual cell models. You'll start with two systems: our Model Performance Dashboard, which benchmarks those models, and the pipeline that turns single-cell data into their training sets. You'll set the technical direction for both and connect them, so every new dataset version is judged by what it does to the models. Later in your first year, the role widens into the engineering that makes the ML team's training and evaluation run fast and at scale, lab-in-the-loop infrastructure, validation frameworks for agent-written software, and Arc's rules for what research data we bring in, keep and share. The role consolidates ownership that is split across several teams today.

Requirements

Bachelor’s degree with 8 years of relevant experience, or Master’s degree with 6 years of relevant experience.
You've owned a production data or ML platform that researchers depend on, end to end (pipelines, storage, versioning), and set its technical direction.
You design and build systems with AI tools and agents, and you make their output verifiable with evals, QC gates or tests before anyone relies on it.
You've learned an unfamiliar technical domain well enough to make design calls in it.
You work directly with stakeholders to find requirements, with no PM in between.
You make the people around you better through code review, pairing and writing patterns down.
You've diagnosed and fixed performance problems in Python or PyTorch systems with profilers and measurements: GPU utilization, memory, data loading.
Work onsite in our Palo Alto office at least 3 days a week.

Nice To Haves

You've worked with single-cell or perturbation data (AnnData/h5ad).
You've built or run model evaluations: benchmarks, eval harnesses, PyTorch/GPU inference.
You've run multi-node distributed training (DDP/FSDP, NCCL, SLURM) or raised a training job's throughput. This is a plus, not a requirement: we care more that you learn it fast with agents and show the numbers moved.
You've led an effort with one or two other engineers.

Responsibilities

Own how Arc measures the performance of its virtual cell models, starting with the backend of our Model Performance Dashboard, which you'll take over within your first two months.
Decide with the ML team which evaluation tools they need, then design and build them so researchers can evaluate and compare models themselves, and measure what each dataset version does to model performance.
Redesign the pipeline that turns single-cell data into training sets for our virtual cell models, then run it with AI agents: agents find, process and curate datasets, and automated quality checks, not manual review, decide whether a dataset version is ready for model training.
Own the lab-in-the-loop infrastructure that turns model performance results into the case for what Arc's labs should test next, working directly with the science leads who design those experiments.
Build or buy validation frameworks (tests, evals, QC gates, policies) that let agent-written code and apps ship without line-by-line human review. Start with our team, where you'll also own how we work with coding agents, then extend them to apps that scientists across Arc build themselves.
Own the engineering that makes the ML team's training and evaluation run fast, reliably and at scale. The ML team owns the research and the models. Once a model is ready to push, it enters the train-and-eval CI system you own, which feeds our Model Performance Dashboard. You'll also make multi-node training a supported path, track down distributed-training failures, and measure and raise throughput and GPU utilization (MFU) on clusters that Arc IT operates.