Staff Engineer - ML Infra / MLOps

Quince•Palo Alto, CA

About The Position

Quince is seeking a Staff ML Engineer to join their growing team. The ideal candidate is a deeply technical ML infrastructure engineer who combines hands-on mastery with system-level thinking. This role involves building and operating production-grade ML systems at scale, including distributed training pipelines, feature stores, and high-throughput inference serving. The engineer will be responsible for engineering platforms that other engineers love to use, designing for extensibility, observability, and resilience. This role is for someone who gravitates toward the hardest problems, such as optimizing GPU utilization, designing zero-downtime model deployment systems, or defining architectural patterns for industrializing AI at scale. The position requires high autonomy, exceptional standards, and the ability to elevate engineers through code reviews, technical mentorship, and by setting a high bar for quality.

Requirements

8+ years of industry experience, with at least 4+ years of focused, hands-on work in ML Infrastructure, MLOps, or large-scale Data Platform engineering.
Proven track record of designing and building MLOps platforms that support the full model lifecycle — from data ingestion and distributed training to real-time inference and model governance.
Deep expertise in cloud-native infrastructure (preferably AWS), Kubernetes (EKS), Docker, and Infrastructure as Code tools (Terraform/Pulumi).
Hands-on mastery of ML frameworks such as PyTorch, TensorFlow, Kubeflow, or SageMaker, with strong opinions on building a cohesive, high-leverage developer experience.
Expertise in building Feature Stores and high-throughput data pipelines (Spark, Flink, Kafka), with a strong understanding of training/serving skew and data consistency.
Expert-level knowledge of CI/CD for ML, including model versioning, experiment tracking, and deployment strategies such as blue-green and canary rollouts.
Demonstrated ability to optimize GPU utilization, implement model batching, and systematically reduce cloud infrastructure costs.
Strong operational instincts, with a history of improving reliability through rigorous on-call practices, proactive monitoring, and root-cause analysis.
You understand the hustle of a startup and are good at handling ambiguity.
You are a curious, quick learner who loves to experiment and thrives at a rapid pace.

Responsibilities

Own the end-to-end technical design of Quince’s ML platform — including model training, serving, feature pipelines, and monitoring — ensuring it is modular, scalable, and built for long-term extensibility.
Design and implement the core developer experience for Quince’s Data Scientists and AI Researchers, enabling them to move from “idea to production” with minimal friction and maximum reliability.
Set and uphold engineering standards in CI/CD for ML, Infrastructure as Code (IaC), model versioning, experiment tracking, and deployment strategies (blue-green, canary) — and build the tooling that makes those standards the path of least resistance.
Lead the technical evaluation and selection of core platform components — from inference runtimes and feature stores to orchestration frameworks — with a clear-eyed view of build vs. buy tradeoffs.
Design and implement GPU utilization optimizations, model batching strategies, and cloud cost controls to maximize performance per dollar across training and inference workloads.
Architect ML serving infrastructure that gracefully handles traffic surges, seasonal spikes, and model version transitions, with robust monitoring, alerting, and automated recovery.
Provide deep technical mentorship to junior and mid-level engineers through design reviews, code reviews, and pairing sessions — raising the collective technical bar without adding process overhead.
Lead root-cause analyses (RCAs) for production failures and drive systemic, permanent fixes over reactive patches.
Model a culture of rigorous on-call discipline and accountability.