Senior Software Engineer, TorchPass Product Engineering

Clockwork.io•Palo Alto, CA

13d

About The Position

TorchPass keeps distributed PyTorch training running through hardware and network failures. The Product Engineering team makes it something customers can install, trust, and run on their own clusters. We are looking for an experienced engineer who has turned working infrastructure software into a product that other people can deploy and operate. You will own the parts of TorchPass that customers touch, and you will be one of the engineers they hear from when something does not work. This is a product and deployment role. It does not require GPU runtime, NCCL, or kernel experience.

Requirements

4+ years building or operating infrastructure software in production
You have taken a system that worked for its authors and made it installable and supportable by people who did not build it
Strong debugging skills on production systems you did not write
Deep experience with Kubernetes or Slurm
Strong Python, and the ability to read and change Go
You use AI coding tools well, and you understand and stand behind the code you ship
You write documentation that people follow successfully
You are comfortable working with customers on technical problems
A degree in computer science, electrical engineering, or a related field, or equivalent experience

Nice To Haves

Experience with GPU clusters or machine learning training infrastructure
Experience with release engineering: build pipelines, packaging, artifact delivery
Experience bringing up clusters across more than one cloud provider

Responsibilities

Build and own the customer-facing surfaces of TorchPass: installation, configuration, and the views that show customers what the product is doing
Keep product documentation accurate with every release
Work directly with customers and our solutions engineers during evaluations: read logs, separate product defects from environment issues, and say clearly what happened
Turn what we learn in the field into fixes, tests, and documentation
Define, together with the core engineering team, what TorchPass needs from a customer's cluster and what it guarantees in return
Extend TorchPass to new schedulers and job launchers as customers need them