Member of Technical Staff - AI Infra Algorithm
About The Position
This individual-contributor role owns the scheduling and routing layer of an AI inference platform, including request admission, prioritization, batching, and placement across a heterogeneous, multi-cloud GPU fleet. The role involves designing foundational systems from the ground up and balancing tail latency, fleet utilization, multi-tenant fairness, and service-level commitments under real production load. The position combines quantitative systems modeling with hands-on engineering and production ownership. The successful candidate will apply queueing theory, control theory, and load-shedding principles; implement and optimize performance-sensitive systems; and carry designs through rollout, monitoring, and live operation. As part of a small team, this person will make foundational architecture decisions, document them clearly, and help raise the engineering standard across the team.
Requirements
- Substantial experience building and operating large-scale distributed systems in production, including ownership of load-bearing systems.
- A track record of designing core systems from zero to one and owning the consequences of architectural decisions.
- Hands-on experience with scheduling, load balancing, request routing, or resource allocation systems.
- Strong systems fundamentals in operating systems, networking, and concurrency, with the ability to reason quantitatively using queueing theory, control theory, or similar methods.
- Fluency in a performance-sensitive language such as Go, Rust, or C++, including profiling and latency optimization in production.
- Comfort with GPU infrastructure and LLM inference fundamentals, including batching, KV cache behavior, and throughput/latency tradeoffs.
- Clear technical writing and the ability to communicate difficult design decisions.
- Uses AI tools daily and has a perspective on how they affect infrastructure development.
Nice To Haves
- Mandarin proficiency.
- Kubernetes and multi-cloud operations experience.
- Open-source contributions to inference, serving, or scheduling projects.
- Experience operating GPU clusters at scale.
- Deep expertise in GPU infrastructure and LLM inference.
Responsibilities
- Own the scheduling and routing layer, designing and building request admission, prioritization, batching, and placement across a heterogeneous GPU fleet spanning multiple clouds and accelerator types.
- Drive down tail latency while improving fleet utilization.
- Apply queueing theory, control theory, and load-shedding principles to make the platform predictable under bursty, multi-tenant traffic.
- Build multi-tenant fairness and isolation so priority guarantees and SLO commitments hold when capacity is saturated.
- Instrument, observe, and debug distributed behavior in production, and carry designs through rollout and real-world load.
- Make foundational architecture decisions, write design documents, and raise the engineering standard of the team.
Benefits
- Meaningful equity.
- Medical coverage.
- Dental coverage.
- Vision coverage.
- Other benefits.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
