Senior Inference Optimization ML Engineer

Rhoda AI • Mountain View, CA

5d

About The Position

Rhoda AI is hiring an Inference Optimization engineer to make its multimodal foundation models run quickly, correctly, and cost-effectively across a broad range of hardware. The role spans AMD Instinct GPUs, Google TPUs, wafer-scale and dataflow accelerators, consumer GPUs such as the RTX 5090, and Jetson/Thor-class compute on robots. The engineer will work closely with research, robotics, and infrastructure teams and help determine which hardware is best suited to each workload.

The position owns inference optimization end to end, from model and kernel optimization to on-robot deployment, hardware evaluation, compression, correctness testing, and benchmarking infrastructure. It also partners with research engineers to help design models that are efficient and portable across hardware. The work has direct influence on accelerator purchasing and deployment decisions in the cloud and on robots.

Requirements

• At least 2 years of experience in inference optimization, ML systems, ML compilers, or a closely related field.
• Deep hands-on experience with PyTorch.
• Strong understanding of compute, memory-bandwidth, memory-capacity, and interconnect bottlenecks and how they differ across hardware architectures.
• Experience with model optimization techniques such as quantization (INT8/FP8/FP4, AWQ), distillation, pruning, and compilation using tools such as torch.compile, XLA, TensorRT, IREE/MLIR, TVM, or vendor compilers.
• Familiarity with inference-serving frameworks and their support for different hardware, such as vLLM on ROCm/TPU, SGLang, JetStream, or Triton Inference Server.
• Excellent debugging and measurement skills, including the ability to identify bottlenecks, design experiments, and validate improvements.
• A high-ownership, pragmatic mindset focused on the total cost of a solution rather than peak benchmark numbers.

Nice To Haves

• Experience optimizing inference for diffusion models.
• Production or deep project experience with an accelerator stack other than NVIDIA datacenter GPUs, or substantial deployment work on consumer GPUs.
• Kernel-authoring experience on multiple backends, such as HIP or Composable Kernel on AMD, Pallas on TPU, Triton on NVIDIA and AMD, or CUTLASS/CuTe.
• Contributions to open-source ML compilers or hardware backends such as PyTorch, OpenXLA, Triton, vLLM, IREE, or TVM.
• Experience running inference on wafer-scale systems, TPU pods, or fleets of consumer GPUs.
• Hands-on experience with Jetson/Thor-class platforms, including TensorRT on Jetson, JetPack, DLA, or power and thermal profiling.
• Experience with multimodal or video model inference, including variable-length sequences and packing or bucketing.
• Experience with speculative decoding, continuous batching, disaggregated prefill/decode, or other LLM-serving optimizations.
• Experience with streaming or low-latency systems for real-time robot control.
• Experience working with hardware vendors on driver or compiler bugs and early-access silicon.

Responsibilities

• Optimize foundation-model inference across vision, video, language, proprioception, and action workloads, improving latency, throughput, and cost per inference.
• Port and optimize models for AMD, TPU, wafer-scale and dataflow accelerators, and consumer Blackwell GPUs, using relevant vendor frameworks, SDKs, and compilers.
• Deploy and optimize models on Jetson/Thor-class embedded compute within latency, power, thermal, and memory constraints, and determine how workloads should be split between robot and cloud.
• Evaluate hardware using cost/performance models covering performance per dollar and watt, memory capacity and bandwidth, and interconnect; provide recommendations for serving, offline evaluation, and on-robot use.
• Develop and tune hardware-portable and vendor-specific kernels and optimizations, including attention, KV cache, memory layout, and fused operators.
• Apply quantization, pruning, distillation, and model or tensor parallelism to fit models to constrained hardware and improve performance.
• Build tests for numerical parity, policy quality, and robot-relevant behavior across hardware backends.
• Build benchmarking and regression infrastructure that tracks latency, throughput, and cost per inference across model versions and hardware targets, with automated regression detection.
• Work with research engineers to design models that are efficient and portable across hardware.

Benefits

• Direct impact on inference cost and scale, with optimization gains lowering the costs of model serving and evaluation and enabling deployment on more robots.
• Strategic influence over hardware selection through benchmarks and recommendations that shape accelerator purchases and deployments in the cloud and on robots.
• High ownership on a small team.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed