(USA) Principal, Software Engineer

Walmart • Sunnyvale, CA

7d • Onsite

About The Position

The Principal Software Engineer on Walmart's Private Cloud Storage Engineering team architects and operates some of the largest Ceph storage clusters in the industry, supporting hundreds of petabytes of mission-critical data. This hands-on technical leader drives the design, scalability, performance optimization, and automation of distributed storage platforms across multiple global production environments. Partnering closely with computer, networking, security, and cloud engineering teams, they deliver highly available storage solutions that meet demanding SLAs for latency, throughput, resiliency, and scalability. The role requires deep expertise in Ceph, Linux internals, OpenStack, and large-scale infrastructure, along with a proven ability to mentor engineers, influence technical strategy, and contribute to open-source storage communities. Walmart’s Private Cloud Storage Engineering team designs and operates some of the largest Ceph storage clusters worldwide, managing extensive data across multiple sites. The team supports critical workloads for business operations, customer platforms, and innovation projects. Working at the convergence of distributed storage, Linux internals, networking, and cloud infrastructure, the team addresses complex challenges in scalability, performance, and resilience. Emphasizing technical expertise, automation, and observability, the team collaborates with engineers across computer, networking, security, and cloud domains to advance enterprise storage solutions at Walmart’s scale.

Requirements

• 15+ years of experience in distributed storage systems and Linux infrastructure engineering, including 10+ years of hands-on expertise designing, deploying, and operating large-scale Ceph environments at petabyte scale.
• Extensive experience developing, enhancing, and supporting Ceph CSI and Cinder CSI drivers to provide persistent storage for cloud-native, containerized platforms.
• Deep knowledge of Linux internals, kernel tuning, storage architectures, and networking technologies, including TCP/IP, BGP, RDMA, BlueStore, RocksDB, OSD architecture, and erasure coding.
• Strong experience troubleshooting and optimizing system performance using tools such as perf, blktrace, tcpdump, bpftrace, and eBPF, with proven expertise in infrastructure automation using Python, Ansible, and Terraform.
• Hands-on experience with Kubernetes, OpenStack (Cinder and Swift), and public cloud platforms, including AWS, Azure, and Google Cloud Platform (GCP).
• Bachelor's degree in computer science, computer engineering, computer information systems, software engineering, or related area and 5 years’ experience in software engineering or related area.
• OR 7 years’ experience in software engineering or related area.

Nice To Haves

• Master’s degree in computer science, computer engineering, computer information systems, software engineering, or related area and 3 years' experience in software engineering or related area.
• Knowledge in implementing Web Content Accessibility Guidelines (WCAG) 2.2 AA standards, assistive technologies, and integrating digital accessibility seamlessly.
• Knowledge of accessibility best practices and joining in creating accessible products and services following Walmart’s accessibility standards and guidelines for supporting an inclusive culture.

Responsibilities

• Architect, deploy, and manage large-scale Ceph storage clusters across global production environments, ensuring high availability, durability, and scalability through advanced CRUSH map design, erasure coding, and replication strategies.
• Design, develop, and maintain Ceph CSI and Cinder CSI drivers to provide persistent, resilient, and high-performance storage services for containerized workloads on Walmart's Cloud Native Platform.
• Troubleshoot and optimize performance across the storage, Linux, and networking stacks, leveraging tools such as perf, blktrace, bpftrace, and tcpdump to identify and resolve complex bottlenecks.
• Build and enhance automation for cluster provisioning, lifecycle management, capacity expansion, monitoring, and operations using Ansible, Terraform, and Python.
• Lead root cause analysis for critical production incidents, mentor engineers on storage architecture and best practices, and contribute to the Ceph and Kubernetes CSI open-source communities.

Benefits

• Medical coverage
• Vision coverage
• Dental coverage
• 401(k)
• Stock purchase
• Company-paid life insurance
• PTO (including sick leave)
• Parental leave
• Family care leave
• Bereavement leave
• Jury duty leave
• Voting leave
• Short-term disability
• Long-term disability
• Company discounts
• Military Leave Pay
• Adoption expense reimbursement
• Surrogacy expense reimbursement
• Walmart-paid education benefit program (Live Better U) for full-time and part-time associates, covering tuition, books, and fees for programs ranging from high school completion to bachelor's degrees, including English Language Learning and short-form certificates.
• Performance bonuses (annual or quarterly)

What This Job Offers

• Job Type: Full-time
• Career Level: Principal