Senior Software Engineer, ML API – AI Inference

Cerebras Systems • Sunnyvale, CA

17h

About The Position

Cerebras is hiring a Senior Software Engineer to build and evolve the ML API layer for its disaggregated AI inference systems, combining GPU-accelerated prefill with decode on the Cerebras Wafer-Scale Engine. The role owns substantial features and components across inference APIs, model integration, and request-routing services, creating a reliable and consistent experience across models and accelerator backends.

This hands-on individual-contributor role spans machine learning systems, API design, model serving, and distributed systems. The engineer translates evolving model capabilities and customer needs into designs, leads implementation across components, and carries features through validation, rollout, and production operation. Areas of focus include streaming, sampling, tool use, structured outputs, multimodal inputs, and model configuration. The role works closely with model enablement, compiler, runtime, cloud infrastructure, product, customer-facing teams, and customers. It also offers opportunities to lead small project teams, mentor engineers, and improve production software quality and reliability.

Requirements

• At least 5 years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
• Strong programming ability in Python and proficiency in Go, C++, Rust, or another language used to build production services.
• Experience designing and delivering features spanning multiple components, documenting designs, evaluating tradeoffs, and coordinating implementation with other engineers.
• Experience building and evolving production APIs, including validation, error handling, compatibility, and testing.
• Familiarity with REST, gRPC, or streaming protocols.
• Experience integrating software across service, framework, runtime, or infrastructure boundaries.
• Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD.
• Ability to diagnose correctness, reliability, and performance issues across distributed components and deliver measurable improvements.
• Strong communication and collaboration skills, including leading technical delivery for a small project and supporting other engineers through reviews or mentorship.
• Ability to turn ambiguous requirements into maintainable software while balancing customer needs, delivery timelines, and operational quality.
• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice To Haves

• Experience building or operating production ML or LLM inference APIs, including OpenAI-compatible interfaces.
• Experience extending or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another ML systems project.
• Understanding of tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, or constrained decoding.
• Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, or inference request scheduling.
• Experience building API conformance, model-quality, numerical-comparison, determinism, or performance-regression test systems.
• Experience with multi-model or multi-tenant serving, including routing, admission control, quotas, rate limiting, or capacity-aware scheduling.
• Experience developing performance-sensitive or highly concurrent services in Go, C++, Rust, or a similar language.
• Familiarity with GPU inference, AMD Instinct accelerators, ROCm, CUDA, collective communication, or multi-node model serving.
• Familiarity with reduced-precision inference and quantization formats such as BF16, FP8, FP4, INT8, or INT4.
• Experience building SDKs, developer tools, configuration systems, or self-service ML workflows, or collaborating directly with customers and open-source communities.

Responsibilities

• Own features end to end by designing, implementing, testing, and operating inference capabilities for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging features.
• Lead multi-component delivery by writing design documents, evaluating tradeoffs, planning milestones, and coordinating implementation with engineers and partner teams.
• Deliver consistent API behavior across GPU prefill, Cerebras decode, and other inference backends; maintain compatibility with widely adopted inference interfaces and apply validation, versioning, and deprecation practices.
• Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features into the serving platform.
• Extend inference services and integrate them with frameworks and runtimes such as vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components.
• Implement routing, state-transfer integration, error handling, retries, cancellation, and request lifecycle management for disaggregated GPU prefill and Cerebras decode.
• Diagnose bottlenecks and improve time to first token, streaming latency, throughput, and resource efficiency through changes to tokenization, serialization, batching, scheduling, and communication.
• Build conformance tests, model-validation suites, numerical comparisons, integration tests, and performance benchmarks to catch regressions and support safe releases.
• Improve service metrics, tracing, logging, alerts, and diagnostic tools; investigate production incidents, resolve root causes, and improve resilience and incident response.
• Partner with product, support, and solutions teams to understand user needs and improve configuration, documentation, examples, and debugging workflows.
• Provide design and code reviews, mentor less-experienced engineers, share domain knowledge, and remove delivery roadblocks.

Benefits

• Job stability with startup vitality.
• A work culture described as simple and non-corporate that respects individual beliefs.
• Opportunities to publish and open source cutting-edge AI research.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior