Senior Inference Optimization ML Engineer

Rhoda AI•Mountain View, CA

About The Position

At Rhoda AI, the Senior Inference Optimization ML Engineer will make the company’s multimodal foundation models run quickly, correctly, and cost-effectively across a wide range of hardware, including AMD Instinct GPUs, Google TPUs, wafer-scale and dataflow accelerators, consumer GPUs such as the RTX 5090, and Jetson/Thor-class robot compute. The role covers end-to-end inference optimization, from choosing appropriate hardware for workloads to improving model performance on each target. The engineer will work closely with research, robotics, and infrastructure teams. The position includes optimizing vision, video, language, proprioception, and action models for latency, throughput, and cost per inference; porting and tuning models across accelerator stacks; and deploying inference on robots under strict latency, power, thermal, and memory limits. It also involves evaluating hardware, building cost/performance models and recommendations, developing portable and vendor-specific kernels, applying model compression, ensuring cross-backend correctness, and maintaining benchmarking and regression infrastructure. The engineer will collaborate with research engineers to help design models that are efficient and portable from the outset. This role offers direct influence over inference costs, model deployment scale, and the accelerators Rhoda selects for cloud and robot use. The engineer will have high ownership of the optimization layer on a small team.

Requirements

At least 2 years of experience in inference optimization, ML systems, ML compilers, or a closely related field.
Deep hands-on experience with PyTorch.
Strong understanding of compute, memory-bandwidth, memory-capacity, and interconnect bottlenecks across hardware architectures.
Experience with model optimization techniques, including quantization such as INT8, FP8, FP4, or AWQ; distillation; pruning; and compilation.
Experience with compilation tools such as torch.compile, XLA, TensorRT, IREE/MLIR, TVM, or vendor compilers.
Familiarity with inference-serving frameworks and their support for different hardware, such as vLLM, SGLang, JetStream, or Triton Inference Server.
Excellent debugging and measurement skills, including the ability to identify bottlenecks, design experiments, and validate improvements.
A high-ownership, pragmatic mindset and focus on total solution cost rather than peak benchmark results.

Nice To Haves

Experience optimizing inference for diffusion models.
Production or substantial project experience with an accelerator stack other than NVIDIA datacenter GPUs, such as AMD ROCm, Google TPU, wafer-scale or dataflow accelerators, Groq, Tenstorrent, AWS Trainium/Inferentia, or Intel Gaudi, or serious experience deploying on consumer GPUs.
Kernel-authoring experience on more than one backend, such as HIP or Composable Kernel on AMD, Pallas on TPU, Triton on NVIDIA and AMD, or CUTLASS/CuTe.
Contributions to open-source ML compilers or hardware backends such as PyTorch, OpenXLA, Triton, vLLM, IREE, or TVM.
Experience running inference on wafer-scale systems, TPU pods, or fleets of consumer GPUs.
Hands-on experience with Jetson/Thor-class platforms, including TensorRT on Jetson, JetPack, DLA, or power and thermal profiling.
Experience with multimodal or video model inference, including variable-length sequences and packing or bucketing.
Experience with speculative decoding, continuous batching, disaggregated prefill/decode, or other LLM-serving optimizations.
Experience with streaming or low-latency systems for real-time robot control.
Experience working with hardware vendors on driver or compiler bugs and early-access silicon.

Responsibilities

Own cost-efficient inference end-to-end and improve foundation-model latency, throughput, and cost per inference.
Port and optimize models for AMD hardware using ROCm, HIP, MIOpen/hipBLASLt, and Composable Kernel.
Port and optimize models for Google TPUs using XLA, JAX, Pallas, and PyTorch/XLA.
Optimize models for wafer-scale and dataflow accelerators using their SDKs and compilers.
Optimize models for consumer Blackwell GPUs, including FP8/FP4, limited memory, and multi-GPU setups over PCIe.
Deploy and optimize models on Jetson/Thor-class embedded compute, accounting for latency, power, thermal, and memory constraints.
Determine how inference work should be divided between robots and the cloud to meet real-time control requirements.
Evaluate hardware using cost/performance models covering performance per dollar, performance per watt, memory capacity and bandwidth, and interconnect.
Provide recommendations on which hardware to buy or rent for serving, offline evaluation, and on-robot use.
Develop and tune portable kernels with technologies such as Triton, Pallas, and MLIR-based compilers, as well as vendor-specific kernels where beneficial.
Optimize attention, KV cache, memory layout, and fused operators across backends.
Apply quantization, pruning, distillation, and model or tensor parallelism to fit and optimize models on constrained hardware.
Build tests for numerical parity, policy quality, and robot-relevant behavior across backends.
Build benchmarking and regression infrastructure to track latency, throughput, and cost per inference across model versions and hardware targets, with automated regression detection.
Work with research engineers to design efficient, portable models and avoid hardware-specific operators where appropriate.