Senior DevOps Engineer, Cloud Simulation Infrastructure

NVIDIA•Santa Clara, CA

1d•$184,000 - $287,500

About The Position

We are seeking a Senior DevOps / Cloud Simulation Infrastructure Engineer to own the complete end-to-end cloud execution pipeline for SimReady assets! This role is critical to our product strategy, enabling us to transition from local, workstation-driven validation to high-scale, automated cloud validation on NVIDIA Cloud Functions (NVCF). You will be responsible for deploying a robust, multi-GPU pipeline that supports structural validation, AI-driven runtime behavioral testing, and automated asset remediation. What you'll be doing: Deployment: Deploy full Isaac Sim runtimes within GPU-aware NVCF containers. Manage container packaging, GPU initialization, and runtime utilities for physics, sensor, and rendering validation. Deploy Structural Validation: Deploy services to validate USD structure and compliance without runtime overhead. Deploy Runtime Validation: Architect scalable execution layers to conduct runtime behavior-based testing (e.g., drop/grasp tests). Deploy rule-based systems or AI based systems for automated pass/fail grading. Deploy Automated Remediation: Develop an AI-based pipeline that intercepts failures, triggers automated asset fixes, and re-validates results to ensure quality standards. Cloud Infrastructure Ownership: Scale execution from single-workstation validation to massive, multi-GPU cloud environments. Optimize for performance, addressing function-to-function networking, gRPC bottlenecks, and in-cluster proxy behavior. Artifact & Evidence Pipeline: Automate the generation of verification videos, thumbnails, feature-level reports, and validation metadata. Ensure all assets are traceable and linked to quality gates. Observability & CI/CD: Establish robust CI/CD, cluster verification, and monitoring pipelines. Implement logging, metrics, and tracing to ensure services are observable, debuggable, and production-ready. Operational Reliability: Implement atomic update semantics and safe failure handling to ensure validation processes never corrupt the primary asset library.

Requirements

BS or MS degree in Computer Science, Computer Engineering, or related field (or equivalent experience).
8+ years of professional experience working on DevOps and/or cloud simulation.
Extensive experience in production-grade DevOps, SRE, or Infrastructure Engineering, with a focus on GPU-backed cloud services.
Proven expertise in container orchestration (Kubernetes/Docker) and CI/CD pipeline development.
Experience with automated testing frameworks, preferably involving AI/ML inference, computer vision, or rule-based validation.
Proficiency in Python and systems scripting for test orchestration and pipeline automation.
Strong ability to design and maintain distributed job lifecycle services (submit/poll/fetch/cancel) and handle asynchronous failure states.
Ability to diagnose and solve distributed network bottlenecks, including gRPC and function-to-function communication.

Nice To Haves

Direct experience deploying services on NVCF (NVIDIA Cloud Functions) or DGX Cloud.
Deep familiarity with Isaac Sim, Omniverse, USD, or Sensor RTX workflows.
Background in robotics simulation, physical AI, or large-scale content creation pipelines.
Experience building "self-healing" or automated remediation workflows.
Experience with cluster verification frameworks, stress testing, and deployment validation at scale.

Responsibilities

Deploy full Isaac Sim runtimes within GPU-aware NVCF containers. Manage container packaging, GPU initialization, and runtime utilities for physics, sensor, and rendering validation.
Deploy services to validate USD structure and compliance without runtime overhead.
Architect scalable execution layers to conduct runtime behavior-based testing (e.g., drop/grasp tests). Deploy rule-based systems or AI based systems for automated pass/fail grading.
Develop an AI-based pipeline that intercepts failures, triggers automated asset fixes, and re-validates results to ensure quality standards.
Scale execution from single-workstation validation to massive, multi-GPU cloud environments. Optimize for performance, addressing function-to-function networking, gRPC bottlenecks, and in-cluster proxy behavior.
Automate the generation of verification videos, thumbnails, feature-level reports, and validation metadata. Ensure all assets are traceable and linked to quality gates.
Establish robust CI/CD, cluster verification, and monitoring pipelines. Implement logging, metrics, and tracing to ensure services are observable, debuggable, and production-ready.
Implement atomic update semantics and safe failure handling to ensure validation processes never corrupt the primary asset library.