Senior Storage Production Engineer - DGX Cloud

NVIDIA•Us, CA

10d

About The Position

Production engineering is a discipline that involves designing, building, and maintaining large-scale production systems with high efficiency and availability. It encompasses various areas, including software and systems engineering practices, storage, data management, and services. Production Engineers are highly specialized and possess expertise in different domains, such as storage architecture, high-performance distributed storage, data management, systems, networking, coding, database management, capacity planning, continuous delivery and deployment, as well as open-source cloud-enabling technologies like Kubernetes, containers, and virtualization. Their responsibilities include ensuring reliable, scalable, high-performance storage solutions, optimizing data placement and access patterns, managing large-scale distributed storage systems, and ensuring low-latency data access for high-performance computing (HPC) and AI/ML workloads. Storage Production Engineers at NVIDIA ensure that our internal and external-facing GPU cloud services meet reliability and uptime goals as promised to the users while enabling developers to make changes to the existing system through careful preparation and planning while keeping an eye on capacity, latency, and performance. This role also requires a mindset focused on automating storage operations, improving data access efficiency, and optimizing storage performance. Much of our software development focuses on optimizing operations through automation, performance tuning, and improving the efficiency of storage and production systems. Since Production Engineers are responsible for the big picture of how our systems interface with each other, we use a breadth of tools and approaches to tackle a broad spectrum of challenges. Practices such as proactive storage performance monitoring, automated fault detection and remediation, scalable data replication strategies, and integration of intelligent caching mechanisms factor into iterative improvements that are key to system reliability and efficiency. What You Will Be Doing: Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity. Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues. Work with AI/ML workloads to optimize storage architectures for low-latency access, efficient caching, and high-throughput performance. Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization. Support storage services before they go live through activities such as system design consulting, developing automation frameworks, capacity management, and launch reviews. Maintain production storage infrastructure by monitoring availability, latency, and system health, leveraging predictive analytics and AI-driven automation. Optimize storage efficiency through compression, deduplication, tiering strategies, and intelligent workload placement. Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques. Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems. Practice sustainable incident response and blameless root cause analysis. Be part of an on-call rotation to support storage and production systems.

Requirements

BS degree or equivalent experience in Computer Science, Storage Systems, or a related technical field with 8+ years of practical experience.
Experience with distributed and high-performance storage solutions, including clustered and parallel file systems, distributed object storage, and enterprise-grade storage systems.
Solid understanding of block, file, and object storage technologies, including their scalability, reliability, and performance characteristics and standard processes.
Experience with storage networking protocols such as NFS, SMB, iSCSI, S3, Fibre Channel, RDMA, and NVMe over Fabrics.
Expertise in algorithms, data structures, complexity analysis, software design, and automating maintenance of large-scale Linux-based storage systems.
Experience in one or more of the following: C/C++, Java, Python, Go, NodeJS, and Bash for storage automation, monitoring, and performance tuning.
Hands-on experience with infrastructure configuration management tools like Ansible, Chef, Puppet, and Terraform for automating storage deployments.
Experience with observability and tracing tools like InfluxDB, Prometheus, Grafana, and the Elastic stack for monitoring storage system health.
You should possess excellent written and oral communication skills, excellent work ethics, a deep sense of teamwork, love to produce quality work and commitment to finishing your tasks every single day.

Nice To Haves

Deep understanding of large-scale distributed storage architectures, replication strategies, and erasure coding techniques.
Experience in capacity planning, performance tuning, and troubleshooting high-throughput storage systems.
Experience with Git, code review, pipelines, and CI/CD for handling infrastructure as code.
Experience in analyzing and improving distributed storage system performance at scale.
Strong debugging skills with a systematic problem-solving approach to identify complex storage issues.
Proven understanding of network protocols, architectures, and troubleshooting techniques, especially as it relates to storage performance, stability, and availability.
Experience using or operating private and public cloud storage solutions based on Kubernetes, OpenStack, or hybrid cloud architectures.
Ability to design and implement automated storage migration, backup, and disaster recovery strategies.
Thrive in collaborative environments and enjoy working with various teams to optimize storage performance.
Flexible in adapting to different working styles and emerging storage technologies.

Responsibilities

Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity.
Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues.
Work with AI/ML workloads to optimize storage architectures for low-latency access, efficient caching, and high-throughput performance.
Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization.
Support storage services before they go live through activities such as system design consulting, developing automation frameworks, capacity management, and launch reviews.
Maintain production storage infrastructure by monitoring availability, latency, and system health, leveraging predictive analytics and AI-driven automation.
Optimize storage efficiency through compression, deduplication, tiering strategies, and intelligent workload placement.
Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques.
Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems.
Practice sustainable incident response and blameless root cause analysis.
Be part of an on-call rotation to support storage and production systems.