Senior Software Engineer, Private Cloud Compute Inference Routing

Apple • Cupertino, CA

4d

About The Position

Apple is seeking a senior software engineer to build next-generation inference routing for Private Cloud Compute (PCC), the platform behind Siri AI and Apple Intelligence. The role involves designing distributed systems that route inference requests across a global fleet of servers, balancing latency, availability, and compute efficiency while preserving PCC’s privacy guarantees. The engineer will develop routing and scheduling strategies that adapt to changing demand, available capacity, and failures while keeping request content encrypted outside trusted compute nodes.

Working with inference and security teams, the engineer will evaluate approaches such as cache-aware request placement and prefill/decode disaggregation; design system architectures; specify custom network protocols; write high-performance code; and develop scaling strategies for services used by hundreds of millions of people. The engineer will own technical decisions from early design through production, using benchmarks and operational evidence to assess tradeoffs, and collaborate with model serving, infrastructure, security, and SRE engineers to turn inference requirements into reliable routing capabilities.

The role is on Apple’s iCloud Platform Engineering team, which builds server systems for services including Push Notifications, iMessage, FaceTime, iCloud, iCloud Private Relay, and Private Cloud Compute. The team’s distributed systems serve hundreds of millions of users across a billion devices worldwide. The team is described as small, highly collaborative, and high impact.

Requirements

• At least 3 years of professional experience designing, building, and operating large distributed systems in production.
• Server development experience in Java, C/C++, C#, Go, or Swift.
• Knowledge of LLM inference serving, including inference request routing, KV-cache placement and reuse (such as prefix-aware routing), prefill/decode disaggregation, and tradeoffs among latency, throughput, and accelerator use.
• Experience with network protocols, application-level networking, and client-server architectures, including request routing, load balancing, or scheduling at scale.
• Experience benchmarking, debugging, and tuning distributed-system performance.
• Knowledge of Unix system internals.
• Experience with cloud deployment and observability technologies such as Kubernetes.
• Bachelor’s degree in an engineering-related field or equivalent experience.

Nice To Haves

• Experience building LLM inference serving or routing systems such as llm-d, NVIDIA Dynamo, vLLM, SGLang, or similar.
• Familiarity with KV-cache-aware routing, prefix caching, prefill/decode disaggregation, and accelerator capacity placement.
• Experience with TLS, X.509 certificate evaluation, or other secure protocols, and awareness of recent developments in standard protocols.
• Understanding of cryptographic protocols, encryption, key management, and privacy-preserving system design, such as Oblivious HTTP or remote attestation.
• Experience leading a substantial engineering project from design through production, resolving technical tradeoffs, and coordinating across teams.
• Ability to explain technical tradeoffs, collaborate across engineering teams, and make progress on problems with incomplete requirements.

Responsibilities

• Design distributed systems that route inference requests across a global fleet of inference servers.
• Develop routing and scheduling strategies that adapt to changing demand, capacity, and failures while balancing latency, availability, and compute efficiency.
• Preserve Private Cloud Compute privacy requirements, including keeping request content encrypted outside trusted compute nodes.
• Evaluate techniques such as cache-aware request placement and prefill/decode disaggregation with inference and security teams.
• Design system architectures, specify custom network protocols, and write high-performance code.
• Build scaling strategies for services used by hundreds of millions of people.
• Own technical decisions from early design through production, evaluating tradeoffs with benchmarks and operational evidence.
• Collaborate with model serving, infrastructure, security, and SRE engineers to deliver reliable inference routing capabilities.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior