Senior DevOps Engineer, IPP Sanity Engineering

NVIDIA • Santa Clara, CA

1d

About The Position

At NVIDIA, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an Nvidian, you’ll be immersed in a diverse, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world. We are looking for a Senior DevOps Engineer to work in IPP (Infrastructure, Planning and Process) Sanity Engineering. IPP is a core software infrastructure organization within Nvidia. This group works with various other groups within Nvidia Software such as Graphics Processors, Mobile Processors, Deep Learning, Artificial Intelligence and Driverless Cars to cater to their infrastructure needs. These cloud services provide almost half a million automated jobs per day on thousands of servers helping with the productivity of thousands of Nvidia's software engineers worldwide. The cloud hosts a heterogeneous mix of machines and devices with various operating systems (Windows/Linux/Android), a multitude of hardware platforms both Nvidia GPUs and Tegra Processors. Are you passionate about distributed infrastructure and looking for a sophisticated workspace and are ready to build the next generation of cloud services for chip bringups, design creative solutions, mine through data to uncover real problems and fix them? We are looking forward to onboard a fun-loving person like you.

Requirements

• Bachelor's or Master's Degree in Computer Science or Software Engineering, or equivalent experience.
• 8+ years of relevant experience.
• Hands-on coding and debug experience (with any including but not limited to Python, JAVA etc.), cross compiling source code on various platforms, triage, root cause and resolve issues in the bringup infrastructure.
• Familiar with maintenance and setup of Linux, Windows (x64 and arm) hosts, VM and container based environments.
• Unix & TCL shell proficiency is expected.
• Experience in MySQL/No-SQL(plus), should be able to write complex queries.
• Experience with version control systems like Perforce, GIT.
• Demonstrable experience working in large scale enterprise production systems.
• 8+ years of development and operations experience required.

Nice To Haves

• Expertise in creating agentic AI to orchestrate GPU fleet management is a plus.
• Strong background with automating bare metal and VM provisioning at large scale.
• Prior knowledge of VM isolation for GPUs, Nvidia Confidential Computing etc is a major plus.
• Experience with public cloud platforms (AWS, GCP, Azure) and virtualization technologies such as VMware, KVM, Docker, and Kubernetes clusters is desirable
• Experience with debugging GPU performance issues, embedded device software development and automation, software driver development and CUDA/TensorRT applications.

Responsibilities

• Lead end to end SW infrastructure bringup execution of new Nvidia GPU product.
• Develop framework and scripts to automate workflows and deployments in the private cloud environment.
• Have a thorough understanding of Nvidia GPU hardware and driver stack, SBIOS, VBIOS and should be able to enhance automation for farm wide updates of firmware.
• Optimize farm utilization of GPU resources by identifying right set of regression test coverage.
• Champion config automation using world-class configuration management & infrastructure automation (IaC) tools like Chef, Puppet, Ansible, Terraform etc.
• Execute on bringup of specialized products used for accelerated computing and secure AI (Nvidia Confidential Computing), in fast-paced and critically important environments.
• Develop extensive monitoring dashboards systems to have fast, reliable and real-time pulse of the various infrastructure subsystems using state of the art telemetry.
• Will be leading a service charter completely, and will be essentially a responsible pilot in charge for the development, monitoring and automation of that infrastructure.
• Automation and performance tuning of regression test frameworks, creation of self healing/automated recovery solutions for multi-geo regression farms.
• Coherent engagement with collaborators and partner teams across organizations to deliver onboarding of new products in CI/CD.
• Implement seamlessly on multiple parallel bringups scoped out in Nvidia Product Bringup landscape.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level