Inference Engineer

General Compute • San Francisco, CA

14h

About The Position

General Compute is an early-stage neocloud company focused on productionizing purpose-built inference hardware. The Inference Engineer will build and own the inference serving layer between compiled, verified models and live customer requests. This includes request scheduling, batching, KV-cache management, autoscaling across the ASIC fleet, and handling production-scale failure modes.

This is a founding role on a small team with broad ownership. The engineer will design the serving architecture and be responsible for reliability, with no separate SRE or platform team to take over those responsibilities. The role focuses on building a serving stack designed specifically for General Compute’s hardware rather than adapting a GPU-first framework, and includes shaping the technical roadmap and helping establish the team’s engineering standards.

Requirements

• At least 5 years of experience building and operating production infrastructure systems, ideally including high-throughput or low-latency serving systems.
• Production experience with LLM inference serving, such as request batching, KV-cache management, or continuous batching; research code alone is not sufficient.
• Experience owning reliability and being on call for an important system, with an approach that designs for failure.
• Strong systems fundamentals, including concurrency, networking, and scheduling, with the ability to reason about performance at the hardware level.
• Self-directed and comfortable working with ambiguity in a founding role.

Nice To Haves

• Experience serving models on non-NVIDIA accelerators, such as TPU, Trainium/Inferentia, Tenstorrent, Groq, or Cerebras.
• Familiarity with serving frameworks such as vLLM, TGI, TensorRT-LLM, or SGLang, and informed views on their limitations.
• Experience running infrastructure at a small company or in a founding or early-engineer role.
• Exposure to capacity planning or fleet management for specialized hardware.

Responsibilities

• Own the inference serving stack end-to-end, including request routing, batching, scheduling, and autoscaling serving replicas for a model.
• Tune batching strategy, KV-cache handling, and hardware utilization to reduce cost per token and improve throughput.
• Build monitoring, alerting, and failover capabilities to support reliable service under customer load, and respond to incidents.
• Define the interface between model bring-up and compilation and production serving, and route issues to the appropriate team.
• Help shape the serving roadmap, including potential work on multi-tenant isolation, speculative decoding, and scheduling strategies.
• Establish technical and code-quality standards as the serving team grows.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed