Inference Technical Lead, On-Device Transformers

OpenAI • San Francisco, CA

39d • Hybrid

About The Position

As a Technical Lead on the Future of Computing Research team, you will work together with both the best ML researchers in the world and the greatest design talent of our generation to push the frontier of model capabilities.
This role is based in San Francisco, CA. We follow a hybrid model with 4 days a week in the office and offer relocation assistance to new employees.
In this role, you will:
Evaluate and select silicon platforms (GPUs, NPUs, and specialized accelerators) for on-device and edge deployment of OpenAI models.
Work closely with research teams to co-design model architectures that meet real-world deployment constraints such as latency, memory, power, and bandwidth.
Analyze and model system performance, identifying tradeoffs between model design, memory hierarchy, compute throughput, and hardware capabilities.
Partner with hardware vendors and internal infrastructure teams to bring up new accelerators and ensure efficient execution of transformer workloads.
Build and lead a team of engineers responsible for implementing the low-level inference stack, including kernel development and runtime systems.
Run through the necessary walls to take nascent research capabilities and turn them into capabilities we can build on top of.
You might thrive in this role if you:
Have experience evaluating or deploying workloads on GPUs, NPUs, or other specialized accelerators.
Understand the performance characteristics of transformer models, including attention, KV-cache behavior, and memory bandwidth requirements.
Have designed or optimized high-performance compute systems, such as inference engines, distributed runtimes, or hardware-aware ML pipelines.
Have experience building or leading teams working on low-level performance-critical software such as CUDA kernels, compilers, or ML runtimes.
Have already spent time in the weeds teaching models to speak and perceive.

Requirements

• Have experience evaluating or deploying workloads on GPUs, NPUs, or other specialized accelerators.
• Understand the performance characteristics of transformer models, including attention, KV-cache behavior, and memory bandwidth requirements.
• Have designed or optimized high-performance compute systems, such as inference engines, distributed runtimes, or hardware-aware ML pipelines.
• Have experience building or leading teams working on low-level performance-critical software such as CUDA kernels, compilers, or ML runtimes.
• Have already spent time in the weeds teaching models to speak and perceive.

Responsibilities

• Evaluate and select silicon platforms (GPUs, NPUs, and specialized accelerators) for on-device and edge deployment of OpenAI models.
• Work closely with research teams to co-design model architectures that meet real-world deployment constraints such as latency, memory, power, and bandwidth.
• Analyze and model system performance, identifying tradeoffs between model design, memory hierarchy, compute throughput, and hardware capabilities.
• Partner with hardware vendors and internal infrastructure teams to bring up new accelerators and ensure efficient execution of transformer workloads.
• Build and lead a team of engineers responsible for implementing the low-level inference stack, including kernel development and runtime systems.
• Run through the necessary walls to take nascent research capabilities and turn them into capabilities we can build on top of.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed