Machine Learning Systems Engineer

Recruiting From Scratch • Palo Alto, CA

20h • Onsite

About The Position

Our client is building a new generation of highly efficient AI models designed to dramatically improve the speed and economics of large language model inference. The company has pioneered diffusion-based language models that generate responses in parallel rather than relying exclusively on traditional sequential token generation. This approach enables significantly faster and more efficient AI inference while maintaining competitive model quality. The company launched one of the first commercially available diffusion-based language models in early 2025 and is now deploying large-scale AI models with Fortune 500 organizations. The team is small, highly technical, and research-driven, with engineers working directly alongside world-class researchers and founders. The organization places a strong emphasis on technical depth, experimentation, performance optimization, and production-scale AI infrastructure. As a Machine Learning Systems Engineer, you'll work on the infrastructure that enables large-scale model training and inference, contributing directly to systems that make advanced AI models faster, more efficient, and more reliable. This is an opportunity to join an elite AI team where you can work at the intersection of machine learning, distributed systems, GPU infrastructure, and high-performance model serving.

Requirements

• 2–5 years of professional experience in ML Systems Engineering, ML Infrastructure, AI Infrastructure, or related engineering roles
• Experience building and operating production ML systems
• Experience working on infrastructure for model training and/or inference
• Experience deploying machine learning models into production environments
• Experience working with GPU-based computing infrastructure
• Experience building scalable ML or distributed systems
• Experience working with modern deep learning frameworks
• Experience operating in technically demanding engineering environments
• Experience collaborating closely with research and engineering teams
• Strong ownership mentality with demonstrated execution ability
• Comfortable working on complex technical problems with limited precedent
• Strong interest in machine learning systems and AI infrastructure
• Ability to operate effectively in a fast-moving, research-driven environment
• Strong Python engineering experience
• Strong experience with PyTorch, TensorFlow, or comparable ML frameworks
• Experience with GPU computing and CUDA
• Experience with ML inference systems such as vLLM, TensorRT, ONNX Runtime, or SGLang
• Strong understanding of model serving and inference optimization
• Experience with Docker and containerized ML workloads
• Experience with Kubernetes
• Experience working with AWS, Azure, or other major cloud platforms
• Experience building distributed systems or scalable infrastructure
• Experience with ML training and inference pipelines
• Strong understanding of performance optimization and system reliability
• Experience debugging production ML infrastructure
• Strong understanding of computer science and software engineering fundamentals
• Ability to reason about GPU utilization, memory constraints, latency, and throughput
• Strong debugging and performance analysis capabilities
• Bachelor's degree or higher in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, or related technical field preferred
• Strong computer science, systems, and machine learning fundamentals
• Equivalent practical engineering experience accepted
• Exceptional technical ownership
• Strong analytical and problem-solving ability
• Deep technical curiosity
• Comfortable working on difficult and ambiguous infrastructure problems
• Strong communication skills
• Comfortable collaborating with researchers and highly technical engineers
• High execution velocity
• Strong attention to system performance and engineering quality
• Bias toward experimentation and continuous improvement
• Low-ego collaborative mentality
• Comfortable receiving and incorporating technical feedback
• Strong ability to reason from first principles
• Comfortable working in a small, high-performing team
• Strong interest in cutting-edge AI systems
• Willingness to work five days per week in Palo Alto

Nice To Haves

• Experience with Kubeflow or comparable ML orchestration platforms preferred
• Advanced degree in Machine Learning, Computer Science, or related field is a plus

Responsibilities

• Design, build, and operate infrastructure supporting large-scale ML training and inference systems
• Develop high-performance systems for serving and deploying large language models
• Optimize model inference for latency, throughput, memory utilization, and cost efficiency
• Build and maintain production ML infrastructure across GPU and cloud environments
• Work with inference engines such as vLLM, TensorRT, ONNX Runtime, and SGLang
• Develop and optimize GPU-accelerated ML workloads using CUDA
• Build scalable training and inference pipelines using PyTorch and/or TensorFlow
• Deploy and manage ML workloads across Kubernetes and containerized environments
• Design distributed systems capable of supporting high-volume model inference
• Improve model serving performance across different hardware and infrastructure configurations
• Build reliable systems for model deployment, monitoring, evaluation, and production operations
• Work closely with research teams to translate new model architectures into production systems
• Optimize infrastructure for emerging diffusion-based language models and other generative AI architectures
• Develop tooling and automation for ML experimentation and deployment
• Build and maintain cloud infrastructure across AWS, Azure, or comparable environments
• Work with Kubeflow and other ML orchestration platforms
• Investigate performance bottlenecks across compute, networking, memory, and model-serving layers
• Develop systems that make model training and inference faster, more efficient, and more reliable
• Contribute to system architecture and technical strategy across the ML infrastructure stack
• Operate with high ownership in a fast-moving, deeply technical AI environment
• Work closely with founders and researchers on highly technical infrastructure challenges

Benefits

• Competitive Equity Package
• Opportunity to work on cutting-edge diffusion-based language models
• Direct collaboration with world-class AI researchers and founders
• Opportunity to work on large-scale ML training and inference infrastructure
• Exposure to advanced GPU optimization and AI systems engineering
• Significant technical ownership in a small, elite engineering organization
• Opportunity to influence foundational ML infrastructure and model-serving architecture
• High-growth AI company environment
• Opportunity to work on AI systems being deployed by Fortune 500 organizations

What This Job Offers

• Job Type: Full-time
• Career Level: Senior