Principal Software Engineer - AKS Flex: High Scale

Microsoft • ,

2d • $142,800 - $304,200

About The Position

Imagine working at the forefront of the cloud industry, redefining where Kubernetes can run. The AKS Flex team is building the next generation of Azure Kubernetes Service: a single AKS control plane that can manage worker nodes running anywhere, whether in other Azure regions, on-premises data centers, bare-metal fleets, edge sites, or other clouds.
We are hiring a Principal Software Engineer to scale this vision. You will lead the scaling of etcd and the Kubernetes control plane so a single cluster can reliably manage far larger, more geographically distributed, and more heterogeneous node fleets than ever before. You will also build the platform behind flex nodes for AKS and Unbounded, our open-source project for joining any Linux machine to a Kubernetes cluster, including provisioning, lifecycle, and the Kubernetes-native APIs that make remote machines manageable at scale.
You will solve deeply technical distributed-systems problems in control plane performance, consistency and durability, node provisioning and lifecycle, security, and reliable production operations. Your work will directly unlock capacity for GPU-hungry AI workloads, data-residency-constrained customers, and organizations that want one operating model for all of their compute.
This is a unique opportunity to define the architecture of a new product area from its early days, represent Microsoft in the open-source community, and shape what hyperscale managed Kubernetes looks like without boundaries.
Microsoft's mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Requirements

• Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, Go, Rust, C++, C#, Java, or Python OR equivalent experience
• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.

Nice To Haves

• Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, Go, Rust, C++, C#, Java, or Python OR Bachelor's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, Go, Rust, C++, C#, Java, or Python OR equivalent experience.
• Deep understanding of etcd internals, including Raft consensus, MVCC, the watch mechanism, compaction and defragmentation, and the bbolt storage backend, as well as experience operating etcd at scale.
• Deep understanding of storage systems and databases, including storage engines (B-trees, LSM trees), write-ahead logging, replication, consistency models, transactions, and performance tuning.
• Experience with Kubernetes control plane internals: API server, watch cache, controller patterns, API priority and fairness, and scalability limits.
• Experience with performance engineering: profiling, load testing, capacity modeling, and diagnosing latency and throughput issues in production.

Responsibilities

• Own the architecture for control plane scalability and the flex nodes and Unbounded platform, and help define the multi-year technical roadmap for AKS Flex in partnership with product and engineering leadership.
• Lead cross-organization design efforts spanning AKS, Azure compute, storage, networking, and identity, driving alignment on interfaces, tradeoffs, and sequencing.
• Make and defend key architectural decisions around correctness, durability, security, scale limits, and failure modes, and establish the design review and quality bar for the area.
• Represent Microsoft in upstream communities (such as Kubernetes SIG Scalability, SIG etcd, SIG API Machinery, and the Unbounded project), driving proposals and building consensus with external maintainers.
• Mentor and grow senior engineers, and raise the technical capability of the broader team.
• Push the limits of cluster scale by improving the performance, reliability, and efficiency of etcd and the Kubernetes API server under large node counts, high object churn, and heavy watch traffic.
• Analyze and optimize control plane bottlenecks such as write amplification, compaction and defragmentation, watch fan-out, list/relist storms, request prioritization, and storage and memory pressure.
• Lead the design and validation of architectural changes to the control plane storage layer, including approaches to partitioning, offloading, and caching state, while preserving correctness and durability guarantees.
• Build scale and performance testing infrastructure, benchmarks, and SLOs that let us measure control plane limits and prevent regressions before they reach customers.
• Build automation for safe etcd operations at fleet scale: backup and restore, member replacement, quorum recovery, upgrades, and repair.
• Design and build the systems that let worker nodes in other Azure regions, on-premises data centers, edge locations, and other clouds securely join an AKS cluster and behave as standard node pools.
• Build node provisioning and day-2 lifecycle automation across diverse substrates: SSH-based bootstrap, cloud API provisioning in response to unschedulable pods, and bare-metal PXE boot with BMC power management and TPM-based attestation.
• Design Kubernetes-native APIs, controllers, and CRDs that make remote machines declaratively manageable with standard tooling, and that scale efficiently as fleets grow.
• Solve the control plane challenges of remote nodes: high and variable latency, intermittent connectivity, node heartbeats and status churn, node identity and authentication, secure bootstrap, and safe upgrades across heterogeneous hardware including GPUs.
• Contribute to and help lead the Unbounded open-source project, including design discussions, code reviews, and community engagement.
• Participate in production operations and on-call rotations, turning operational learnings into durable engineering improvements.
• Partner across AKS and Azure teams to deliver secure, compliant, and resilient infrastructure.
• Drive engineering culture through technical design, code reviews, and setting standards for reliability and operational excellence across the area.

Benefits

• Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay

What This Job Offers

• Job Type: Full-time
• Career Level: Principal