Engineering Manager, Private Cloud Compute Inference Routing

Apple•Seattle, WA

19h

About The Position

Lead the Private Cloud Compute (PCC) inference-routing organization within Apple’s iCloud Platform Engineering team. The role guides systems that route inference requests across a global, multi-region fleet while maintaining low latency, high availability, efficient accelerator utilization, and PCC privacy guarantees. Request content must remain encrypted outside trusted compute nodes. Set architectural direction for routing and scheduling, working with inference and security teams to evaluate approaches such as cache-aware request placement and prefill/decode disaggregation. The scope includes the inference AI gateway, which authenticates, meters, prioritizes, and streams requests without exposing their content. In the first year, the manager is expected to grow a high-performing organization, establish a technical roadmap, and measurably improve routing latency, availability, and fleet utilization. This work supports Siri AI and Apple Intelligence.

Requirements

Extensive software engineering experience and a demonstrated record of leading teams that design, build, and operate large-scale distributed systems in production, through direct management, indirect oversight, or both.
Ability to recruit, develop, and retain high-performing engineers and leaders.
Experience delivering API gateways, preferably inference AI gateways, request routing, load balancing, scheduling, or traffic management systems at scale.
Expertise in LLM inference serving, including request routing, KV-cache placement and reuse, prefill/decode disaggregation, and tradeoffs among latency, throughput, and accelerator use.
Record of shipping highly reliable systems in high-availability environments, including leading incident management, on-call, and reliability practices.
Technical depth to contribute meaningfully to architecture and design discussions with senior engineers.
Server development background in Java, C/C++, Go, or Swift.
Ability to drive alignment and execution across teams in complex, matrixed environments.
Bachelor’s degree in an engineering discipline or equivalent professional experience.

Nice To Haves

Leadership in developing LLM inference serving or routing platforms such as llm-d, NVIDIA Dynamo, vLLM, SGLang, or managed inference services.
Experience building or operating inference AI gateways and core behaviors such as token-based rate limiting and usage metering, multi-model routing with fallback, retries and circuit breaking, and streaming response handling.
Track record of scaling teams and organizations during rapid growth while maintaining high hiring and engineering standards.
Experience delivering capacity tiers, reservations, or QoS for high-volume workloads and improving accelerator utilization through cross-region, capacity-aware routing.
Understanding of privacy-preserving system design, including TLS, X.509, key management, Oblivious HTTP, and remote attestation.
Experience designing and maturing observability practices, including SLO/SLI frameworks, distributed tracing, and on-call culture.
Experience with container orchestration systems such as Kubernetes.
Ability to build psychologically safe, high-trust team environments that attract and retain top talent.
Experience formulating multi-year roadmaps with product, research, and senior leadership in large organizations that prioritize quality and reliability.

Responsibilities

Lead PCC inference routing and scheduling across a global, multi-region compute fleet.
Guide architectural direction for routing systems that adapt to demand, capacity, and failures while maintaining low latency and high accelerator utilization.
Design routing and scheduling strategies within PCC’s privacy model, keeping request content encrypted outside trusted compute nodes.
Work with inference and security teams to evaluate cache-aware request placement and prefill/decode disaggregation.
Oversee the inference AI gateway layer, including request authentication, metering, prioritization, and streaming responses without exposing request content.
Recruit, develop, and retain high-performing engineers and leaders.
Grow a high-performing organization and establish a clear technical roadmap.
Improve routing latency, availability, and fleet utilization.
Lead operational practices for highly reliable systems, including incident management, on-call, and reliability practices.
Drive alignment and execution across teams in complex, matrixed environments.