Engineering Manager, Private Cloud Compute Inference Routing

Apple•Seattle, WA

About The Position

Apple’s iCloud Platform Engineering organization builds the server stack behind services including Push Notifications, iMessage, FaceTime, iCloud, iCloud Private Relay, and Private Cloud Compute (PCC). The Engineering Manager will lead PCC inference routing, guiding systems that direct inference requests to compute resources across a global, multi-region fleet while maintaining low latency, high availability, efficient accelerator utilization, and PCC’s privacy guarantees. This is a technically grounded leadership role responsible for architectural direction, operational practices, and developing engineers and engineering leaders. The manager will lead routing and scheduling strategy within PCC’s privacy model, where request content remains encrypted outside trusted compute nodes. The role includes the inference AI gateway layer, which authenticates, meters, prioritizes, and streams requests using metadata without exposing request content, and involves evaluating techniques such as cache-aware request placement and prefill/decode disaggregation with inference and security teams. In the first year, the manager is expected to grow a high-performing organization, establish a clear technical roadmap, and measurably improve routing latency, availability, and fleet utilization. The role partners with Siri AI, Apple Intelligence, infrastructure, security, model-serving, and Site Reliability Engineering teams to deliver integrated routing capabilities.

Requirements

Extensive software engineering experience and a demonstrated track record leading teams that design, build, and operate large-scale distributed systems in production, through direct management, indirect oversight, or both.
Proven ability to recruit, develop, and retain high-performing engineers and leaders.
Experience delivering API gateways, preferably inference AI gateways, request routing, load balancing, scheduling, or traffic-management systems at scale.
Expertise in LLM inference serving, including request routing, KV-cache placement and reuse, prefill/decode disaggregation, and trade-offs among latency, throughput, and accelerator use.
Track record of shipping highly reliable systems in high-availability environments, including leading incident management, on-call, and reliability practices.
Strong technical depth to contribute meaningfully to architecture and design discussions with senior engineers, with a server-development background in Java, C/C++, Go, or Swift.
Ability to drive alignment and execution across teams in complex, matrixed environments.
Bachelor’s degree in an engineering discipline or equivalent professional experience.
Proven leadership in developing LLM inference serving or routing platforms, such as llm-d, NVIDIA Dynamo, vLLM, SGLang, or managed inference services.
Experience building or operating inference AI gateways and core behaviors such as token-based rate limiting and usage metering, multi-model routing with fallback, retries and circuit breaking, and streaming response handling.
Track record of scaling teams and organizations during rapid growth while maintaining high hiring and engineering standards.
Experience delivering capacity tiers, reservations, or QoS for high-volume workloads, and improving accelerator utilization through cross-region, capacity-aware routing.
Understanding of privacy-preserving system design, including TLS, X.509, key management, Oblivious HTTP, and remote attestation.
Experience designing and maturing observability practices, including SLO/SLI frameworks, distributed tracing, and on-call culture.
Experience with container orchestration systems such as Kubernetes.
Demonstrated ability to build psychologically safe, high-trust team environments that attract and retain top talent.
Experience formulating multi-year roadmaps in collaboration with product, research, and senior leadership within large organizations that prioritize quality and reliability.

Responsibilities

Recruit, develop, and lead engineers and engineering managers responsible for PCC inference routing and request-scheduling services, fostering an inclusive culture.
Establish technical strategy with senior engineers and guide architectural decisions and trade-offs in global traffic management, capacity-aware placement, load balancing, and failover.
Translate organizational objectives into actionable roadmaps, execution plans, and measurable results.
Oversee adaptive routing, admission control, prioritization, and overload management to maintain responsiveness and fairly share capacity across workloads.
Evolve inference AI gateway capabilities, including authentication, token-aware rate limiting and quotas, model and version routing, fallback and retries, and streaming response delivery, while preserving end-to-end encryption through metadata-based operation.
Collaborate with model-serving and infrastructure leaders to assess prefill/decode disaggregation and cache-locality optimizations within PCC’s privacy architecture.
Develop and scale operations through observability, service-level objectives, testing, safe deployment, and incident management in partnership with SRE.
Improve efficiency through benchmarking and capacity-aware scheduling, using queueing delay, latency, throughput, and utilization to guide investment.
Provide senior leadership with transparent updates on team health, technical risk, and strategic trade-offs, translating complex technical issues into actionable decisions.
Partner with Siri AI, Apple Intelligence, infrastructure, security, and SRE teams to deliver well-integrated routing capabilities.