Senior Inference Engineer

Uniphore•Palo Alto, CA

1d•$200,000 - $350,000

About The Position

Uniphore is seeking a Senior or Staff Inference Engineer to own the inference layer of its enterprise AI platform. The role combines model-inference expertise with systems engineering to deliver a context-aware runtime serving fine-tuned small language models across cloud and on-premises environments and widely used GPU platforms. This is a hands-on individual-contributor position working closely with research, model engineering, knowledge systems, agent, and platform teams. The engineer will build context-aware model execution; optimize KV caches, computation, and inference performance; manage model and adapter lifecycles; improve production reliability; and support portable deployments across NVIDIA and AMD GPUs and single- or multi-GPU configurations.

Requirements

5+ years of engineering experience, including 3+ years of hands-on experience with GPU systems, distributed inference, or ML serving.
Production experience with vLLM, SGLang, or TensorRT-LLM.
Strong Python and C++ (C++17/20) skills and ability to read and debug CUDA C++ or HIP kernels.
Hands-on GPU profiling and performance optimization experience.
Expertise in model inference internals, including autoregressive generation, prefill versus decoding, attention architectures, compute- versus memory-bound execution, and numerical precision.
Knowledge of speculative decoding, including draft-and-verify execution, acceptance/rejection sampling, adaptive speculation, and throughput tradeoffs.
Knowledge of GPU kernels, including FlashAttention-style kernels, tiling, operator fusion, and memory access patterns.
Proficiency in CUDA, HIP, Triton, or CUTLASS.
Knowledge of KV and prefix caching, including paged KV storage, fragmentation, cache quantization, offloading, cross-request prefix reuse, and tenant isolation.
Knowledge of context-window management, including token budgeting, context selection, prompt compression, conversation compaction, sliding windows, and sparse attention.
Knowledge of serving and scaling, including continuous batching, chunked prefill, admission control, quantization, multi-GPU parallelism, and prefill/decode disaggregation.

Nice To Haves

Custom GPU kernel development using CUDA, HIP, Triton, or CUTLASS.
PyTorch internals, including torch.compile, custom operators, MoE, or disaggregated prefill/decode.
LoRA/SLM adapter serving and zero-downtime hot swapping.
Production experience with NVIDIA and AMD Instinct hardware.
Telemetry experience with DCGM, AMD SMI, Prometheus, and Grafana.

Responsibilities

Build a context-aware runtime that integrates context selection, compression, conversation compaction, and persistent task state with model execution; adapt context budgets, model selection, and reasoning effort to each task.
Optimize KV-cache and computation through paged KV-cache and prompt-prefix-cache management, cache-aware routing, retention, eviction, and offloading while preserving correctness as context, permissions, model versions, and adapters change.
Implement and tune speculative decoding, attention kernels, continuous batching, quantization, chunked prefill, and multi-GPU parallel execution.
Build model and adapter lifecycle management, multi-adapter serving, zero-downtime hot swapping, and resource scheduling.
Implement production observability using DCGM, Prometheus, and Grafana, as well as tenant isolation, overload protection, failure recovery, and safe rollout and rollback.
Build portable runtime integrations across NVIDIA and AMD GPU platforms, cloud and on-premises environments, and single- or multi-GPU configurations.