ML Infrastructure Engineer - New Grad

SambaNova • San Jose, CA

2d

About The Position

SambaNova is hiring a new-grad ML Infrastructure Engineer for its ML Infrastructure team, which builds and operates the inference stack serving SambaNova models on RDU accelerators. The role involves writing production Python for the inference serving stack, including the vLLM integration and public APIs. The engineer will help bring inference features to production on the RDU and develop accuracy and regression tests to verify those features before release.

Requirements

• B.S. in Computer Science, Electrical Engineering, or a related field.
• Strong programming skills in Python.
• Understanding of algorithms, data structures, concurrency, and systems design.
• Coursework or project experience in distributed systems or machine learning.
• Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM.
• Experience with inference techniques such as speculative decoding, KV caching, or continuous batching.
• Experience with PyTorch.

Nice To Haves

• M.S. or Ph.D. in Computer Science, Electrical Engineering, or a related field.
• Experience building or maintaining APIs or backend services.
• Internship or research experience in ML systems or backend infrastructure.
• Experience with disaggregated systems such as prefill/decode.
• Open source project ownership or contributions.

Responsibilities

• Implement inference features on the RDU, including speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference.
• Adapt vLLM and related serving frameworks to the RDU architecture.
• Build and maintain endpoints in the public inference API.
• Write accuracy verification and regression tests for inference features.
• Debug latency, throughput, and reliability issues in production services.

Benefits

• Equity and benefits as part of a competitive total rewards package.
• 95% premium coverage for employee medical insurance and 77% premium coverage for dependent medical insurance.
• Health Savings Account (HSA) with employer contribution.
• Dental, vision, short- and long-term disability, basic life, voluntary life, and AD&D insurance plans.
• Flexible Spending Account (FSA) options for health care, limited purpose, and dependent care.
• Headspace subscription for employees and dependents.
• Gympass+ membership with access to physical gyms.
• One Medical membership.
• Counseling services through an Employee Assistance Program.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level