Staff Inference Engineer

Uniphore•Palo Alto, CA

1d•$200,000 - $350,000

About The Position

Uniphore is seeking a Senior or Staff Inference Engineer to own the inference layer of its enterprise AI platform. The role combines model inference expertise with systems engineering to deliver a context-aware runtime serving fine-tuned small language models across cloud and on-premises environments and widely used GPU platforms. This is a hands-on individual contributor position working closely with research, model engineering, knowledge systems, agent, and platform teams. The engineer will integrate context selection, compression, conversation compaction, and persistent task state with model execution, adapting context budgets, model selection, and reasoning effort to each task. The role also involves optimizing KV-cache and computation, implementing inference performance improvements, managing model and adapter lifecycles, and building production reliability and cross-environment deployment capabilities.

Requirements

At least 5 years of engineering experience, including at least 3 years hands-on with GPU systems, distributed inference, or ML serving.
Production experience with vLLM, SGLang, or TensorRT-LLM.
Strong Python and C++ (C++17/20) skills and ability to read and debug CUDA C++ or HIP kernels.
Hands-on GPU profiling and performance optimization experience.
Expertise in model inference internals, including autoregressive generation, prefill versus decoding, attention architectures, compute- versus memory-bound execution, and numerical precision.
Knowledge of speculative decoding, including draft-and-verify execution, acceptance/rejection sampling, adaptive speculation, and throughput tradeoffs.
Knowledge of GPU kernels, including Flash Attention-style kernels, tiling, operator fusion, and memory access patterns.
Proficiency with CUDA, HIP, Triton, or CUTLASS.
Knowledge of KV and prefix caching, including paged KV storage, fragmentation, cache quantization, offloading, cross-request prefix reuse, and tenant isolation.
Knowledge of context-window management, including token budgeting, context selection, prompt compression, conversation compaction, sliding window, and sparse attention.
Knowledge of serving and scaling, including continuous batching, chunked prefill, admission control, quantization, multi-GPU parallelism, and prefill/decode disaggregation.

Nice To Haves

Custom GPU kernel development using CUDA, HIP, Triton, or CUTLASS.
PyTorch internals, including torch.compile, custom operators, MoE, or disaggregated prefill/decode.
LoRA/SLM adapter serving and zero-downtime hot swapping.
Production experience with NVIDIA and AMD Instinct hardware.
Telemetry experience with DCGM, AMD SMI, Prometheus, and Grafana.

Responsibilities

Build a context-aware runtime integrating context selection, compression, conversation compaction, persistent task state, and model execution.
Adapt context budgets, model selection, and reasoning effort to each task.
Build paged KV-cache and prompt-prefix-cache management, including cache-aware routing, retention, eviction, and offloading.
Preserve correctness as context, permissions, model versions, and adapters change.
Implement and tune speculative decoding, attention kernels, continuous batching, quantization, chunked prefill, and multi-GPU parallel execution.
Build model and adapter lifecycle management, multi-adapter serving, zero-downtime hot swapping, and resource scheduling.
Implement observability, tenant isolation, overload protection, failure recovery, and safe rollout and rollback.
Build portable runtime integrations across NVIDIA and AMD GPU platforms, cloud and on-premises environments, and single- or multi-GPU configurations.