Senior Staff Software Engineer, AI Runtime
About The Position
As a Senior Staff Software Engineer on Databricks’ AI Runtime team, you will work on the systems that power a serverless platform for GPU training, solving difficult technical problems and helping shape the product’s future. AI Runtime enables customers to train models alongside their enterprise data and supports large-scale workloads, including training and post-training models with hundreds of billions to nearly a trillion parameters. You will drive the architecture and long-term technical direction of AI training infrastructure, establish reliability and observability strategy for a growing fleet, and improve the experience of developers and AI agents using the platform. The role spans multi-node training, orchestration, GPU scheduling, data loading, checkpointing, and performance efficiency across model architectures and GPU accelerators. You will collaborate with research, product, and customers to set priorities and align teams on technical decisions, while championing engineering excellence and mentoring and growing engineers.
Requirements
- 15+ years of experience in large-scale distributed systems, with significant depth in GPU training, high-performance computing, or ML systems.
- Hands-on experience training large models with FSDP, DeepSpeed, or Megatron, and fluency in parallelism strategies.
- Expertise in GPU performance, including accelerator architecture, high-speed interconnects, and kernels.
- Strong understanding of training resilience patterns, such as checkpointing, failure detection, and automatic recovery.
- Background building and operating multi-tenant cloud platforms as products.
- Track record of setting technical direction across teams and delivering business impact.
- Ability to bring teams to agreement on difficult trade-offs.
- Reputation as a mentor sought out by senior engineers.
- BS in Computer Science or a related field.
Nice To Haves
- MS or PhD in Computer Science or a related field.
Responsibilities
- Drive AI Runtime architecture and lead long-term technical direction in AI training infrastructure.
- Define reliability and observability strategy for a fleet growing in scale and complexity.
- Shape the developer experience for people and agents using AI Runtime.
- Anticipate industry trends and select technical investments for new GPU generations and workload types.
- Solve complex multi-node training problems, including orchestration, GPU scheduling, data loading, and checkpointing.
- Improve training speed and cost by increasing efficiency across model architectures and GPU accelerators.
- Work with research, product, and customers to determine product priorities and align teams on technical decisions.
- Champion engineering excellence, raise the engineering bar, and help hire and develop engineers.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
