Senior Engineering Manager, AI Infrastructure

The Allen Institute for Artificial Intelligence • Seattle, WA

8h • $146,880 - $220,320 • Onsite

About The Position

We are seeking a Senior Manager, AI Infrastructure to run the day-to-day operation of the systems that power our research. Reporting to the VP of Engineering, you will own the execution and reliability of our high-performance computing (HPC) environment which includes on-prem GPU clusters and the software orchestration layer that schedules workloads across a hybrid cloud environment. This is a hands-on operational leadership role: your mandate is to keep the platform fast, reliable, and well-utilized, and to deliver against the roadmap set with your PM counterpart.

Requirements

• 12+ years in infrastructure, systems engineering, or HPC (or an advanced degree with 8+ years), including 2+ years supervising a small engineering team (5+).
• Bachelor's degree in a related field: a relevant advanced degree may substitute for equivalent years of technical work experience.
• Direct experience operating large-scale NVIDIA GPU clusters and high-performance networking (InfiniBand/RoCE).
• Strong background in Kubernetes, Slurm, or similar orchestration frameworks, particularly in hybrid-cloud configurations.
• Hands-on experience with distributed filesystems (e.g., WEKA, Ceph, Lustre) and cloud storage integration at scale.
• Proficient in designing and managing SDLC processes including sprint planning and technical design reviews.
• Proficient in Go or Python.
• Deep, hands-on understanding of the Linux kernel, container runtimes, and distributed systems.
• Understanding of the performance implications of InfiniBand topologies and NCCL optimizations.
• Comfortable making trade-offs between technical elegance and operational necessity.
• Ability to triage and mitigate immediate risks, and know when to handle something yourself versus escalate.

Responsibilities

• Manage the availability, performance, and health of our dense on-prem GPU clusters. Coordinate with hardware vendors and internal teams to keep physical infrastructure meeting the demands of frontier model training.
• Operate and improve Beaker, our internal orchestration platform by optimizing resource allocation and driving high utilization across on-prem assets and elastic cloud resources (AWS/GCP).
• Execute and continuously improve our storage environment, balancing high-throughput performance for active training against cost-effective durability for petascale research data. Contribute to the longer-term storage roadmap.
• Manage GPU compute allocation against budget. Track utilization, surface the data, and recommend when to burst to the cloud versus investing in on-prem capacity, escalating larger trade-offs as needed.
• Serve as the technical bridge to our research teams. Ensure infrastructure is an accelerator, not a bottleneck, for a diverse set of research objectives.
• Manage and grow a team of systems engineers, SREs, and software developers. Set the bar for operational rigor, engineering quality, and a collaborative culture, and keep the team unblocked and delivering.

Benefits

• Medical, dental, vision, and an employee assistance program.
• Health savings account plan.
• Healthcare reimbursement arrangement plan.
• Health care and dependent care flexible spending account plans.
• Company’s 401k plan.
• $125 per month to assist with commuting or internet expenses.
• $200 per month for fitness and wellbeing expenses.
• Up to ten sick days per year.
• Up to seven personal days per year.
• Up to 20 vacation days per year.
• Twelve paid holidays throughout the calendar year.
• Annual bonuses.
• Long-term incentive plan.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior