Software Engineer, ML Infra
About The Position
We're hiring a Software Engineer to sit at the day-to-day interface between research and infrastructure. This is a generalist role with broad scope: you'll be one of the people always in the room for infra decisions on the ML systems side, with a clear enough view of upcoming compute needs to see support burden coming before it arrives. You'll also be one of the people who babysits hero runs — the ones at 2am when a 4k-GPU job hits a weird Xid and someone needs to decide, quickly and correctly, whether to drain the node, restart the job, or escalate to NVIDIA before the run loses checkpoints. That kind of judgment, built across the kernel, the network, the scheduler, and the application layer, is the core of the job.
Requirements
- Credible, hands-on competence in 4 or more of the following: Linux kernel, networking, GPUs / CUDA, distributed systems runtimes, storage, compilers / language runtimes, observability internals — at this level, this required range is the qualifying signal
- Comfort operating without a clearly defined owner, and the judgment to know when to dig in yourself versus when to escalate
Nice To Haves
- Track record of being the person other engineers escalate to, across 2+ companies
- Shipped meaningful contributions in 3+ distinct technical stacks
- Track record of leading major incidents where the root cause was non-obvious
- Researcher-facing comfort: can talk to a researcher about their workload without making them feel dumb, and can tell them no when the right answer is no
- Experience operating at the scale of frontier training or inference clusters, and appetite for owning the hardest, least-defined problems in that stack
Responsibilities
- Debug across the full stack — kernel, NCCL, scheduler, application, and telemetry — often in the same afternoon, to find root causes that don't show up in any single layer
- Provide embedded, hands-on support during hero runs and major incidents, staying with a problem until it's genuinely resolved
- Serve as the front door for researchers when something's broken and it isn't obvious who owns it
- Lead postmortems and build the tooling that prevents the next incident — acting as a force multiplier, not just a responder
- Mentor other engineers into this kind of cross-stack breadth
Benefits
- generous health, dental, and vision benefits
- unlimited PTO
- paid parental leave
- relocation support
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
