Senior AI Inference Engineer

StackYak•,

2d•Remote

About The Position

StackYak is building the infrastructure layer for AI. We are an early-stage, funded company building software that brings compute, GPU infrastructure, networking, and inference together into one product. The opportunity is large, the market is moving quickly, and we are building for real production workloads from the start. This is not internal IT. This is not a slow-moving infrastructure team maintaining someone else's platform. The infrastructure is the product. We are a small, senior team with very little bureaucracy. This is a founding role in its discipline. You will be the first person here whose primary responsibility is the systems layer beneath the product, and the shape it takes will largely be yours to decide. Treat this document as a starting point rather than a boundary. The people who do well here take ground early and are not asked to give it back. We move quickly. We do not have months for someone to learn the fundamentals of their discipline. You should already be very good at what you do, be able to ramp into adjacent areas quickly, and be comfortable operating without perfect requirements or neatly defined boundaries. We need someone who can take a model, a set of GPUs, and a production requirement and determine how that model should actually run. Hand you a frontier open-weight model — dense or mixture-of-experts, tens to hundreds of billions of parameters — along with the hardware actually available and the latency a customer actually needs, and you should be able to make the calls that follow: precision, quantization, tensor and pipeline and expert parallelism, memory strategy, batching, concurrency, topology, and serving runtime. Then defend them, measure them, and operate them. Doing that well is the baseline. What makes the role interesting is the question sitting underneath it. Serving expertise is scarce, slow to acquire, and currently lives in the heads of a small number of people who have done it enough times to have the instinct. We do not accept that it has to stay that way. A great deal of what those people know is reasoning, and reasoning can be written down, tested, and eventually carried out by something other than a person at two in the morning. How far that can be pushed is genuinely open, and you would be one of the people finding out. This is not a research position and it is not an architecture-only role. You will benchmark, deploy, debug, tune, automate, and operate real inference systems — and then carry them in production.

Requirements

Run large language models in production, under real load, with someone depending on them.
Served a model across multiple GPUs, and across multiple nodes.
Sized a model against hardware and been right about whether it would fit.
Chosen a quantization and precision strategy with a real quality and cost consequence attached to the choice.
Tuned batching and concurrency past the point where the easy wins ran out.
Measured latency, TTFT, throughput, utilization, and cost — and defended the numbers to someone motivated to disbelieve them.
Worked in NVIDIA and/or AMD inference environments.
Written Python that other people run in production: tooling and product, not scripts.
Debugged Linux and systems problems below the container boundary.
Found the root cause of an inference failure that was not in the serving runtime.

Nice To Haves

You have run inference infrastructure at an AI company, inference provider, GPU cloud, hyperscaler, or serious internal AI platform.
You have worked across multiple GPU generations and vendors, and can say concretely how that changed your decisions.
You understand distributed inference and the networking implications of multi-node serving.
You have benchmarked and compared serving runtimes rather than treating one framework as the answer to every problem.
You can explain why a deployment is configured the way it is instead of repeating a vendor recipe.
You have automated deployment decisions, or built schedulers, placement systems, capacity planners, or similar infrastructure.
You build and test things outside your assigned roadmap because you want to know how they actually work.

Responsibilities

How models run. Placement, parallelism, precision, and memory strategy across single-GPU, multi-GPU, and multi-node — and the reasoning behind each choice, in a form someone other than you can audit.
The serving runtimes. vLLM, SGLang, TensorRT-LLM, and whatever displaces them: which one we use where, and what we do when the one we picked turns out to be wrong.
What we are willing to promise. Concurrency limits, latency and TTFT targets, throughput under load. You own the evidence behind a number before it becomes a commitment to a customer.
Benchmarking as an institution. Methodology, harness, reproducibility — and the standing to say that a result does not mean what someone wants it to mean.
Model lifecycle in production. Loading, startup, health, upgrades, capacity, and failure handling, all of it happening while customers are connected.
Multi-tenant versus dedicated serving, and where the line between them actually falls.
Inference incidents, including the ones where the serving runtime turns out to be innocent and the cause is GPU, driver, host, or network.
Turning all of the above into software, so that it stops being institutional memory with a single point of failure.