Principal Engineer, Model Performance

Runware • San Francisco, CA

15h • Remote

About The Position

Runware is building and operating its own end-to-end AI inference infrastructure. Its Sonic Inference Engine has served more than 20 billion requests for over 1 million developers and 500 million end users, with more than 400,000 models. The company is scaling its Sonic Inference Pods and distributed GPU fleet across the US and Europe.

As Principal Engineer, Model Performance, you will own the performance and cost of inference across the fleet, covering diffusion and video generation, speech, 3D, and LLM workloads across models and GPU generations. You will set the performance roadmap, personally build and tune critical components, and establish and lead the Model Performance team. The role combines hands-on frontier inference engineering with technical leadership, hiring, mentoring, and cross-functional work with inference, infrastructure, and hardware teams, as well as NVIDIA and open-source communities.

You will determine and implement the optimizations that improve latency, cost, throughput, and quality, and bring them into production on real traffic. Success includes bringing newly released open models to a strong latency-cost frontier within days, steadily reducing cost per request with each hardware generation, and building a team able to deliver this work independently.

Requirements

• 10+ years of experience in GPU, HPC, or deep learning systems engineering, including several years optimizing inference performance for production models at a company operating at scale.
• A track record of shipping performance improvements, including taking a state-of-the-art model from a reference implementation to the latency-cost Pareto frontier on real hardware.
• Current, hands-on expertise in CUDA and GPU architecture, including kernel design, memory hierarchy, occupancy, Tensor Cores, fused kernels, CUDA Graphs, NCCL, and Nsight Systems and Nsight Compute.
• Production experience with at least one major inference engine, such as TensorRT-LLM, vLLM, or SGLang, and modern serving techniques including continuous batching, paged KV cache, speculative decoding, prefill/decode disaggregation, quantization, and tensor, pipeline, and expert parallelism.
• Fluency in Python and C++, comfort with PyTorch internals, torch.compile, and Triton, and willingness to implement benchmarks, kernels, and CI jobs directly.
• Experience leading engineers, including building or running a performance team, hiring, setting technical direction, and remaining hands-on.
• Regular use of AI to accelerate work such as kernel search, profiling analysis, benchmarking, documentation, and code review, with a clear perspective on how a small AI-enabled team should operate.
• Ability to own outcomes in a fast-moving, ambiguous environment, make decisions with incomplete information, and be accountable for latency, cost, and quality across a large model catalogue.

Nice To Haves

• Diffusion and video generation performance experience, including DiT and UNet pipelines, long-sequence attention, step distillation, caching across denoising steps, LoRA hot-swapping, and ComfyUI, Diffusers, or custom-engine loaders and samplers.
• Contributions to open-source inference projects such as TensorRT-LLM, vLLM, SGLang, FlashAttention, CUTLASS, Triton, or Dynamo, or ownership of a widely used open-source project.
• Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and multi-node inference over InfiniBand or RoCE.
• Mixture-of-experts serving experience, including expert parallelism, placement, load balancing, and associated memory and communication trade-offs.
• Experience evaluating optimized-model accuracy using suitable harnesses, datasets, and statistical methods to establish equivalence to original models.
• Experience with non-NVIDIA accelerators, such as AMD, TPU, or custom silicon, and porting inference stacks across them.
• Speech and audio model serving experience, including streaming ASR and TTS with tight latency budgets.
• Published work, patents, or talks on inference performance.

Responsibilities

• Own inference performance end to end across the fleet, including kernels, serving engines, model-level optimizations, and benchmarks for image, video, audio, 3D, and LLM workloads on every GPU generation.
• Set the model performance roadmap and prioritize models, hardware, and techniques based on real traffic, cost, and future pod requirements.
• Personally write and tune CUDA kernels; fuse attention and MLP paths; introduce quantization formats such as FP8, FP4, and INT4 while maintaining quality; and develop parallelism and batching strategies for multi-GPU nodes.
• Build a pipeline to take newly released open models from weights to optimized, production-grade deployment in days, and continue reducing the deployment gap.
• Choose and shape the serving stack, including TensorRT-LLM, vLLM, SGLang, custom diffusion and video pipelines, and the Sonic Inference Engine runtime; decide what to adopt, fork, or build, and contribute upstream where valuable.
• Own reproducible, hardware-aware benchmarks and performance regression systems for throughput, latency, cost, and quality, integrating them into CI to prevent regressions from reaching production pods.
• Work with infrastructure and hardware engineering on GPU memory, storage, and network bottlenecks, and feed findings into future server and pod designs.
• Build and lead the Model Performance team by hiring principal and senior engineers, setting technical standards, mentoring team members, and enabling a small team to serve a large model catalogue.
• Use AI tools and LLM-based agents for kernel search and tuning, benchmark generation, profiling triage, and documentation.

Benefits

• Meaningful stock options.
• Remote-first work, with in-person meetings twice a year.
• Core collaboration hours, with flexibility to set your schedule outside those hours.
• Fast, intense release cycles followed by downtime, which employees are expected to take.
• Generous paid time off, including vacation, sick days, and public holidays.
• Paid maternity, paternity, and caregiver leave.
• Twice-yearly company retreats in inspiring locations.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed