LLM Inference Engineer

Majestic Labs • Los Altos, CA

3d

About The Position

Majestic Labs is a fast-moving US-Israeli AI startup building full-stack custom AI servers for large-scale inference and training. In this role, the LLM Inference Engineer bridges custom silicon and production-grade AI by owning the end-to-end LLM serving stack on Majestic hardware, from serving APIs through KV cache management, batching, and scheduling. The main focus is porting leading inference frameworks such as vLLM and SGLang to the accelerator and optimizing them for performance, throughput, batch sizes, and context lengths. The engineer will identify performance bottlenecks and share findings with kernel, compiler, and hardware teams to guide future development.

Requirements

• At least 5 years of experience building or operating production LLM inference and serving systems.
• Deep, hands-on experience with a modern inference framework such as vLLM, SGLang, TensorRT-LLM, Fireworks, or similar, including its scheduler, paged attention/KV cache, model executor, and backend integration points.
• Strong Python and C++ skills, with the ability to work fluidly in both.
• Understanding of transformer inference, including the prefill/decode split, KV cache behavior, and how batching dynamics affect latency and throughput.
• Distributed inference experience with tensor and pipeline parallelism across multiple devices.
• Ability to profile an end-to-end stack and trace performance regressions from the serving API down to the kernel.

Responsibilities

• Bring up and adapt a modern inference framework, such as vLLM or SGLang, to run on Majestic hardware.
• Develop and optimize runtime components, including continuous batching, the scheduler, paged KV cache, and prefill/decode disaggregation.
• Build distributed inference across accelerators using tensor, pipeline, and expert parallelism, integrated with the collective communication library (CCL).
• Develop the multi-modal pipeline, including image, audio, and video preprocessing, encoder integration, and mixed-modality batching.
• Implement inference-time techniques such as speculative decoding, prefix caching, and structured decoding.
• Profile and benchmark the end-to-end serving path, identify bottlenecks, and feed findings back to kernel, compiler, and hardware teams.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed