Senior Software Engineer, Datacenter Automation Platforms
About The Position
Collaborate with NVIDIA professionals responsible for building platforms that operate AI factories at scale. The team drives product roadmaps, architecture, cross-functional alignment, and implementation across open-source and internal-facing products. It partners with executive leadership on platform strategy and adoption of innovative software development approaches, mentors engineering talent, and turns successful domain solutions into reusable platform capabilities used across the organization. The role builds self-service SDKs, workflow templates, reference implementations, and catalogs of reusable agents, tools, and connectors. It translates operational intent into versioned, testable policies and defines common contracts and standards for tools, state, events, workflow execution, versioning, and compatibility. The engineer guides migrations to shared services and selects runtime and durable execution architectures. The role also builds agents that investigate failures using hardware diagnostics, logs, metrics, topology, configuration, and incident history; propose evidence-backed diagnoses; and create executable remediation plans. It automates datacenter fleet operations, integrates with Kubernetes, fleet management systems, schedulers, and hardware interfaces, and addresses reliability challenges at fleet scale. The engineer establishes shared evaluation, CI/CD, staged rollout, and observability capabilities and drives adoption through reviews, migration plans, documentation, and hands-on cross-team collaboration.
Requirements
- At least 8 years of experience in software engineering or a related field.
- BS in Computer Science, Engineering, Physics, Mathematics, or a comparable field, or equivalent experience.
- Expert-level knowledge of a systems programming language such as Go, Python, and/or Rust.
- Solid understanding of data structures and algorithms.
- Strong background in Linux system administration and management.
- Experience handling cluster orchestration tools, including Kubernetes and/or SLURM.
- Understanding of software development patterns dedicated to agents and current security practices for agentic systems.
- Understanding of performance, security, and reliability in complex distributed systems, including system-level architecture, data synchronization, fault tolerance, and state management.
Nice To Haves
- Experience with open-source software projects, licensing, and governance.
- Proficiency in understanding and managing large-scale distributed systems independently of cloud providers.
- Deep knowledge of datacenter operations and GPU hardware.
- Advanced hands-on experience with cluster management systems such as Kubernetes or SLURM.
- Hands-on experience in Machine Learning Operations, Training, and/or Security fields.
Responsibilities
- Build self-service SDKs, workflow templates, reference implementations, and catalogs of reusable agents, tools, and connectors.
- Prioritize automation based on customer impact, frequency, operational risk, and reduction in human effort.
- Translate operational intent into versioned, testable policies defining triggers, prerequisites, permitted actions, approval requirements, maintenance windows, capacity limits, and recovery or blocking-issue paths.
- Define common contracts for tools, state, events, and workflow execution; establish versioning and compatibility standards; guide migration to shared services; and select runtime and durable execution architectures.
- Build agents that correlate hardware diagnostics, logs, metrics, topology, configuration, and incident history to investigate failures, propose evidence-backed diagnoses, and produce executable remediation plans.
- Enforce authorization independently of model output, retain action histories, and support human review, bounded retries, cancellation, and compensating actions.
- Automate provisioning validation, node quarantine, workload-aware maintenance, repair, upgrades, and return to service.
- Integrate automation with Kubernetes, fleet management systems, relevant schedulers, and hardware interfaces.
- Address fleet-scale challenges such as partial failures, duplicate events, stale state, conflicting remediations, noisy telemetry, control-plane bottlenecks, and correlated rack or cluster failures.
- Establish shared evaluation, CI/CD, staged rollout, and observability capabilities across teams.
- Drive cross-team adoption through build reviews, migration plans, documentation, and hands-on partnership.
- Align infrastructure, networking, storage, security, and workload teams on shared priorities and contribution standards.
- Mentor engineering talent and turn successful domain solutions into reusable platform capabilities.
Benefits
- Eligible for equity and benefits.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
