(USA) Principal, Software Engineer
About The Position
The Principal Software Engineer on Walmart’s Private Cloud Storage Engineering team architects and operates large-scale Ceph storage clusters supporting hundreds of petabytes of mission-critical data. This hands-on technical leader drives the design, scalability, performance optimization, and automation of distributed storage platforms across global production environments. The role partners with computer, networking, security, and cloud engineering teams to deliver highly available storage solutions that meet demanding latency, throughput, resiliency, and scalability requirements. Walmart’s Private Cloud Storage Engineering team designs and operates large Ceph storage clusters across multiple sites, supporting business operations, customer platforms, and innovation projects. The team works across distributed storage, Linux internals, networking, and cloud infrastructure, emphasizing technical expertise, automation, observability, and collaboration. The position requires deep expertise in Ceph, Linux internals, OpenStack, and large-scale infrastructure, as well as the ability to mentor engineers, influence technical strategy, and contribute to open-source storage communities.
Requirements
- 15+ years of experience in distributed storage systems and Linux infrastructure engineering, including 10+ years of hands-on experience designing, deploying, and operating large-scale Ceph environments at petabyte scale.
- Extensive experience developing, enhancing, and supporting Ceph CSI and Cinder CSI drivers for persistent storage on cloud-native, containerized platforms.
- Deep knowledge of Linux internals, kernel tuning, storage architectures, and networking technologies, including TCP/IP, BGP, RDMA, BlueStore, RocksDB, OSD architecture, and erasure coding.
- Strong experience troubleshooting and optimizing system performance using perf, blktrace, tcpdump, bpftrace, and eBPF.
- Proven expertise in infrastructure automation using Python, Ansible, and Terraform.
- Hands-on experience with Kubernetes, OpenStack (Cinder and Swift), and public cloud platforms, including AWS, Azure, and Google Cloud Platform.
- Bachelor’s degree in computer science, computer engineering, computer information systems, software engineering, or a related area and 5 years of experience in software engineering or a related area; or 7 years of experience in software engineering or a related area.
Nice To Haves
- Master’s degree in computer science, computer engineering, computer information systems, software engineering, or a related area and 3 years of experience in software engineering or a related area.
- Background creating inclusive digital experiences, with knowledge of WCAG 2.2 AA standards, assistive technologies, digital accessibility best practices, and Walmart’s accessibility standards and guidelines.
Responsibilities
- Architect, deploy, and manage large-scale Ceph storage clusters across global production environments, ensuring high availability, durability, and scalability through CRUSH map design, erasure coding, and replication strategies.
- Design, develop, and maintain Ceph CSI and Cinder CSI drivers for persistent, resilient, high-performance storage services for containerized workloads on Walmart’s Cloud Native Platform.
- Troubleshoot and optimize performance across storage, Linux, and networking stacks using tools such as perf, blktrace, bpftrace, and tcpdump.
- Build and enhance automation for cluster provisioning, lifecycle management, capacity expansion, monitoring, and operations using Ansible, Terraform, and Python.
- Lead root cause analysis for critical production incidents.
- Mentor engineers on storage architecture and best practices.
- Contribute to the Ceph and Kubernetes CSI open-source communities.
Benefits
- Competitive pay
- Performance-based bonus awards
- Medical coverage
- Vision coverage
- Dental coverage
- 401(k)
- Stock purchase
- Company-paid life insurance
- Paid time off, including sick leave
- Parental leave
- Family care leave
- Bereavement leave
- Jury duty leave
- Voting leave
- Short-term disability
- Long-term disability
- Company discounts
- Military Leave Pay
- Adoption and surrogacy expense reimbursement
- PTO and/or PPTO that may be used for vacation, sick leave, holidays, or other purposes
- Live Better U education benefit program, including tuition, books, and fees for eligible programs
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
