Senior Inference Engineer

Uniphore•Palo Alto, CA

About The Position

Uniphore is seeking a Senior or Staff Inference Engineer to own the inference layer of its enterprise AI platform. This hands-on individual contributor role combines model inference expertise with systems engineering to deliver a context-aware runtime serving fine-tuned small language models across cloud and on-premises environments and commonly used GPU platforms. The engineer will work closely with research, model engineering, knowledge systems, agent, and platform teams. The role involves integrating context selection, compression, conversation compaction, and persistent task state with model execution, and adapting context budgets, model selection, and reasoning effort to tasks. It also focuses on optimizing KV caches and computation, tuning inference performance, managing model and adapter lifecycles, improving production reliability, and building portable deployments across NVIDIA and AMD GPUs and different deployment configurations.

Requirements

At least 5 years of engineering experience, including at least 3 years of hands-on experience with GPU systems, distributed inference, or ML serving.
Production experience with vLLM, SGLang, or TensorRT-LLM.
Strong Python and C++ (C++17/20) skills and ability to read and debug CUDA C++ or HIP kernels.
Hands-on GPU profiling and performance optimization experience.
Knowledge of autoregressive generation, prefill versus decoding, attention architectures, compute- versus memory-bound execution, and numerical precision.
Knowledge of speculative decoding, including draft-and-verify execution, acceptance/rejection sampling, adaptive speculation, and throughput tradeoffs.
Knowledge of GPU kernel concepts such as FlashAttention-style kernels, tiling, operator fusion, and memory access patterns.
Knowledge of KV and prefix caching, including paged KV storage, fragmentation, cache quantization, offloading, cross-request prefix reuse, and tenant isolation.
Knowledge of context-window management, including token budgeting, context selection, prompt compression, conversation compaction, sliding windows, and sparse attention.
Knowledge of serving and scaling techniques, including continuous batching, chunked prefill, admission control, quantization, multi-GPU parallelism, and prefill/decode disaggregation.

Nice To Haves

Custom GPU kernel development using CUDA, HIP, Triton, or CUTLASS.
PyTorch internals, including torch.compile, custom operators, MoE, or disaggregated prefill/decode.
LoRA/SLM adapter serving and zero-downtime hot swapping.
Production experience with NVIDIA and AMD Instinct hardware.
Telemetry experience with DCGM, AMD SMI, Prometheus, and Grafana.

Responsibilities

Integrate context selection, compression, conversation compaction, and persistent task state with model execution.
Adapt context budgets, model selection, and reasoning effort to each task.
Build paged KV-cache and prompt-prefix-cache management, including cache-aware routing, retention, eviction, and offloading.
Preserve correctness as context, permissions, model versions, and adapters change.
Implement and tune speculative decoding, attention kernels, continuous batching, quantization, chunked prefill, and multi-GPU parallel execution.
Build model and adapter lifecycle management, multi-adapter serving, zero-downtime hot swapping, and resource scheduling.
Implement production observability, tenant isolation, overload protection, failure recovery, and safe rollout and rollback.
Build portable runtime integrations across NVIDIA and AMD GPU platforms, cloud and on-premises environments, and single- or multi-GPU configurations.
Work closely with Uniphore’s research, model engineering, knowledge systems, agent, and platform teams.