Senior ML Infrastructure Engineer
About The Position
SambaNova’s ML Infrastructure team builds and operates the inference stack serving its models on RDU accelerators, from request scheduling and caching through the public APIs in SambaStack and SambaCloud. The team takes inference techniques from prototype to production and owns the accuracy infrastructure used to verify shipped features, working closely with ML, compiler, runtime, and product teams. The Senior Software Engineer, ML Infrastructure designs, builds, and operates production-grade inference infrastructure for SambaNova’s Reconfigurable Dataflow Unit (RDU) architecture. The role turns advanced inference techniques into reliable, high-throughput, low-latency services for customers through SambaStack and SambaCloud. It owns systems spanning request scheduling, decoding algorithms, caching layers, API surfaces, and accuracy infrastructure, and partners with other engineering and product teams to deliver features from prototype to production.
Requirements
- B.S. in Computer Science, Electrical Engineering, or a related field.
- 5+ years of industry experience building and operating large-scale distributed systems, ideally in ML serving.
- Strong software engineering fundamentals, including algorithms, data structures, concurrency, and systems design.
- Experience designing and maintaining production services with strict latency, throughput, and availability requirements.
- Working knowledge of modern LLM inference techniques and familiarity with open-source serving stacks such as vLLM, TensorRT-LLM, or SGLang.
- Proficiency in Python.
- Experience collaborating across teams to deliver complex, system-level engineering solutions.
Responsibilities
- Design and productionize advanced inference techniques on RDU to optimize performance and cost, including speculative decoding, constrained decoding, function/tool calling, prompt caching, and long-context inference.
- Own SambaNova’s integration with vLLM and adjacent serving frameworks, adapting them to RDU architecture.
- Own the public inference API surface exposed through SambaStack and SambaCloud.
- Build and maintain accuracy verification and regression infrastructure that gates inference features shipped to customers.
- Partner with ML, compiler, runtime, and product teams to take inference features from prototype to production.
- Contribute to technical design discussions, code reviews, and architectural decisions as a senior individual contributor.
Benefits
- Equity.
- Employee medical insurance with 95% premium coverage.
- Dependent medical insurance with 77% premium coverage.
- Health Savings Account (HSA) with employer contribution.
- Dental insurance.
- Vision insurance.
- Short- and long-term disability insurance.
- Basic Life, Voluntary Life, and AD&D insurance plans.
- Flexible Spending Account (FSA) options for health care, limited purpose, and dependent care.
- Headspace subscription.
- Gympass+ membership with access to physical gyms.
- One Medical membership.
- Counseling services through an Employee Assistance Program.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
