Staff Software Engineer, AI Inference Gateway

Salad

1d•$180,000 - $220,000•Remote

About The Position

SaladCloud runs inference on thousands of geodistributed workstation and consumer GPUs — NVIDIA and AMD — that nobody else can use. Sitting in front of that fleet is a unique AI Gateway written in Rust, built on Pingora and WireGuard, that serves real-time and batch inference and embeddings to customers as a single, reliable API. You'll own it. This is an infrastructure role with real autonomy. You'll design and ship the systems that decide how requests are fulfilled, when to add or drop capacity, and how to bill for every token — and you'll be the person who knows whether a new model, quantization, or GPU is actually worth running.

Requirements

Strong production Rust experience, ideally on high-throughput networked services
Hands-on experience running LLM inference servers (vLLM, llama.cpp, TGI, TensorRT-LLM, or similar) — you know what a KV cache is and why it fills up
Distributed systems fundamentals: load balancing, backpressure, failure handling, unreliable nodes
Comfort operating what you build: metrics, tracing, on-call instincts
Willing to be on-call for a system you helped make quiet — we treat after-hours pages as a design bug to fix, not a lifestyle
Clear written and verbal communication; you can run a design without hand-holding and bring others along

Nice To Haves

Pingora, Tokio, or proxy/gateway internals
Experience with heterogeneous or consumer-grade GPU fleets, CUDA, or ROCm
Quantization formats (GGUF, AWQ, GPTQ, FP8) and their performance characteristics

Responsibilities

Own the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers
Build and tune the fleet-efficiency algorithms — autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar
Operate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers
Design and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already rides on this telemetry, so you'll keep that integration accurate
Spend roughly 25% of your time benchmarking new models, quantizations, and GPU hardware, and work with the product and marketing teams to make the call on what goes into production
Collaborate on cross-system design and integrations with the rest of Salad engineering; explain trade-offs clearly to both technical and non-technical teammates