Staff Inference Engineer

Uniphore•Palo Alto, CA

21h

About The Position

Uniphore is hiring a hands-on individual contributor to own the inference layer of its enterprise AI platform, where agents operate across voice, video, text, and structured data in real time. The role combines model inference expertise with systems engineering to deliver a context-aware runtime serving fine-tuned small language models across cloud and on-premises environments and widely used GPU platforms. The engineer will work closely with research, model engineering, knowledge systems, agent, and platform teams. The position involves building context-aware model execution, optimizing KV-cache and computation, improving inference performance, managing model and adapter lifecycles, and delivering production reliability. It also includes developing portable runtime integrations for NVIDIA and AMD GPUs, cloud and on-premises deployments, and single- or multi-GPU configurations.

Requirements

At least 5 years of engineering experience, including at least 3 years of hands-on experience with GPU systems, distributed inference, or ML serving.
Production experience with vLLM, SGLang, or TensorRT-LLM.
Strong Python and C++ (C++17/20) skills, with the ability to read and debug CUDA C++ or HIP kernels.
Hands-on GPU profiling and performance optimization experience.
Expertise in model inference internals, including autoregressive generation, prefill versus decoding, attention architectures, compute- versus memory-bound execution, and numerical precision.
Knowledge of speculative decoding, including draft-and-verify execution, acceptance/rejection sampling, adaptive speculation, and throughput tradeoffs.
Knowledge of GPU kernels, including Flash Attention-style kernels, tiling, operator fusion, and memory access patterns.
Knowledge of KV and prefix caching, including paged KV storage, fragmentation, cache quantization, offloading, cross-request prefix reuse, and tenant isolation.
Knowledge of context-window management, including token budgeting, context selection, prompt compression, conversation compaction, sliding window, and sparse attention.
Knowledge of serving and scaling, including continuous batching, chunked prefill, admission control, quantization, multi-GPU parallelism, and prefill/decode disaggregation.

Nice To Haves

Custom GPU kernel development with CUDA, HIP, Triton, or CUTLASS.
PyTorch internals, including torch.compile, custom operators, MoE, or disaggregated prefill/decode.
LoRA/SLM adapter serving and zero-downtime hot swapping.
Production experience with NVIDIA and AMD Instinct hardware.
Telemetry experience with DCGM, AMD SMI, Prometheus, and Grafana.

Responsibilities

Integrate context selection, compression, conversation compaction, and persistent task state with model execution.
Adapt context budgets, model selection, and reasoning effort to each task.
Build paged KV-cache and prompt-prefix-cache management, including cache-aware routing, retention, eviction, and offloading.
Preserve correctness as context, permissions, model versions, and adapters change.
Implement and tune speculative decoding, attention kernels, continuous batching, quantization, chunked prefill, and multi-GPU parallel execution.
Build model and adapter lifecycle management, multi-adapter serving, zero-downtime hot swapping, and resource scheduling.
Implement observability, tenant isolation, overload protection, failure recovery, and safe rollout and rollback.
Build portable runtime integrations across NVIDIA and AMD GPU platforms, cloud and on-premises environments, and single- or multi-GPU configurations.