ML Infrastructure Engineer - New Grad

SambaNova Systems • San Jose, TX

1d

About The Position

SambaNova is a leader in next-generation AI infrastructure, delivering a full-stack inference platform for customers worldwide. Its RDU (Reconfigurable Dataflow Unit) uses a dataflow architecture, and its rack-scale SambaRack systems help customers deploy AI models with improved performance, energy efficiency, and time to value.

The ML Infrastructure team builds and operates the inference stack serving SambaNova models on RDU accelerators. The team works on request scheduling, caching, decoding techniques, public APIs, and accuracy testing. As a new grad ML Infrastructure Engineer, you will write production Python for the inference serving stack, including its vLLM integration and public APIs. You will help bring inference features into production on the RDU and develop accuracy and regression tests.

Requirements

• B.S. in Computer Science, Electrical Engineering, or a related field.
• Strong programming skills in Python.
• Understanding of algorithms, data structures, concurrency, and systems design.
• Coursework or project experience in distributed systems or machine learning.
• Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM.
• Experience with inference techniques such as speculative decoding, KV caching, or continuous batching.
• Experience with PyTorch.

Nice To Haves

• M.S. or Ph.D. in Computer Science, Electrical Engineering, or a related field.
• Experience building or maintaining APIs or backend services.
• Internship or research experience in ML systems or backend infrastructure.
• Experience with disaggregated systems, including prefill/decode.
• Open-source project ownership or contributions.

Responsibilities

• Implement inference features on the RDU, including speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference.
• Adapt vLLM and related serving frameworks to the RDU architecture.
• Build and maintain endpoints in the public inference API.
• Write accuracy verification and regression tests for inference features.
• Debug latency, throughput, and reliability issues in production services.

Benefits

• Equity and benefits as part of a competitive total rewards package.
• 95% premium coverage for employee medical insurance and 77% premium coverage for dependent medical insurance.
• Health Savings Account (HSA) with employer contribution.
• Dental, vision, short- and long-term disability, basic life, voluntary life, and AD&D insurance plans.
• Flexible Spending Account (FSA) options for health care, limited purpose, and dependent care.
• Headspace subscription.
• Gympass+ membership with access to physical gyms.
• One Medical membership.
• Counseling services through an Employee Assistance Program.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level