Senior Staff Inference Performance Engineer

d-Matrix • Santa Clara, CA

1d • $175,000 - $265,000

About The Position

At d-Matrix, the Inference Cloud Engineering team builds deployment and orchestration technology for inference services, including an internal cloud for testing and tuning models and a customer-facing cloud with self-service inference endpoints and compute. The Senior Staff Inference Performance Engineer owns end-to-end performance of deployed workloads, from request arrival through result delivery. The goal is for workloads on the orchestrated platform to perform at least as well as they do on bare metal. Current workloads include speculative decoding and attention-FFN disaggregation across d-Matrix Corsair accelerators and GPUs, with additional heterogeneous architectures planned. The role uses a profiling-first method to investigate unfamiliar systems, test hypotheses, and improve performance. The engineer owns the performance outcome, fixing issues directly where appropriate and providing evidence-based handoffs for issues owned by other teams.

Requirements

• At least 8 years of hands-on performance or systems engineering experience, including resolving difficult regressions that cross multiple stack layers.
• A track record of finding performance overhead through profiling in unfamiliar, sparsely documented systems.
• Deep systems fundamentals, including Linux internals, NUMA, CPU pinning and cpusets, PCIe, memory and cache behavior, and network paths.
• Experience running performance-sensitive workloads on Kubernetes, identifying performance costs from scheduler defaults, network overlays, or orchestration abstractions, and closing the resulting gap.
• Working knowledge of LLM inference serving, including batching, KV-cache, prefill/decode, quantization, and speculative decoding.
• Proficiency in Python and C/C++ sufficient to instrument, debug, and patch code across the stack.
• Clear technical writing that hardware, firmware, or platform teams can act on without context from the candidate’s part of the stack.
• Sound judgment about ownership, with examples of correctly handing off problems that were not theirs to fix.
• Fluency with AI and agentic tools for investigation, code search, log triage, and drafting instrumentation, with discipline to verify output against real data.

Nice To Haves

• Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM.
• Hands-on experience building or tuning speculative decoding, disaggregated serving, or heterogeneous serving architectures.
• Experience tuning KV-cache management, request and batch scheduling, or weight prefetch and residency in an LLM serving system.
• GPU programming and profiling experience, such as CUDA, Nsight, or NCCL.
• Experience with pre-production or bring-up hardware, or in organizations where root causes routinely cross team boundaries.
• Rust experience.
• Startup experience or comfort in environments where priorities shift quickly.

Responsibilities

• Diagnose performance problems across the full serving path, including orchestration, inference internals, drivers and runtime, interconnect, host OS and NUMA, and accelerators.
• Isolate problems to a specific system layer and owner, determine whether to fix or route them, and use workarounds and tracked handoffs when appropriate.
• Write minimal, reproducible problem statements that enable cross-stack owning teams to act without repeating the investigation.
• Protect throughput, latency, and stability targets for orchestrated workloads by identifying and closing gaps against bare-metal performance.
• Assess requirements for scaling from single deployments to fleet operation, including routing, placement, and shared-resource contention.
• Capture workload node requirements, including CPU placement, memory and NUMA locality, and network path, in orchestration configuration so performant deployment is the default.
• Build lightweight telemetry and benchmarks that can reach into prototype stacks to isolate or hand off problems.
• Turn recurring investigations into playbooks and mentor engineers in the profiling-first method.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed