Principal Core Infrastructure Engineer - GPU Validation

Oracle•Nashville, TN

4d•Onsite

About The Position

Join Oracle’s HPC team to develop automated tools and validation frameworks that ensure GPU racks are ready for production. The role builds NCCL performance benchmarks, GPU infrastructure and NVLink connectivity tests, Slurm-based validation workflows, and automated FDT and cable diagnostics. It designs, implements, and optimizes distributed-system components with an emphasis on scalability, resiliency, and operability; delivers features and load and performance tests; uses data-plane platforms and distributed-state tools for high-volume retrieval, storage, and processing; and reviews implementations for scalability compliance. The position builds fault-tolerant paths using redundancy, replication, and automatic failover, and applies recovery-oriented principles such as retries, circuit breakers, and timeouts. It proactively detects and mitigates issues using tests, alarms, dashboards, and telemetry; authors runbooks; and participates in incident response and root-cause investigations. It also implements replication and synchronization, develops automation and infrastructure-as-code tools for troubleshooting and maintenance, applies security controls, and follows change, compliance, and documentation standards. The employee must live in or be willing to relocate to a location that allows regular attendance at Oracle’s Nashville, Tennessee office. Oracle lists a U.S. hiring range of $114,600 to $234,600 per annum. The role may be eligible for bonus, equity, and compensation deferral.

Responsibilities

Design and develop scalable validation frameworks for GPU racks and HPC infrastructure.
Build automated benchmarks and diagnostics for NCCL performance, GPU health, NVLink connectivity, InfiniBand, and high-speed networks.
Develop Slurm-based workflows for distributed validation and performance testing.
Implement retries, timeouts, and recovery mechanisms to improve framework resilience.
Develop functional, load, and fault-injection tests to assess reliability and production readiness.
Build dashboards, telemetry, and alerts to monitor infrastructure health and validation results.
Develop automated tools for Linux diagnostics, log analysis, hardware fault isolation, FDT, and cable validation.
Turn incident findings into reusable diagnostics and preventive automation.
Develop recovery runbooks and contribute to incident response and root-cause investigations.
Develop and maintain Bash, Python, and infrastructure-as-code tools for deployment and configuration.
Implement access controls and secure handling of infrastructure credentials and diagnostic data.
Automate patching, updates, and rollback workflows in accordance with change-management practices.
Maintain technical documentation and improve automation coverage across infrastructure workflows.
Translate project requirements into development plans and deliverables.
Manage assigned work with minimal supervision, meeting agreed timelines and quality standards.
Adapt priorities as requirements, resources, and dependencies change.
Work with HPC, infrastructure, and engineering teams to define requirements and align on outcomes.
Use logs, metrics, and diagnostic results to investigate issues and develop lasting solutions.
Resolve issues independently, escalate when needed, and document findings.
Build expertise in GPU systems, HPC technologies, and software development through training and feedback.
Recommend and implement improvements to validation coverage, reliability, and development efficiency.
Share reusable tools and best practices, and incorporate team feedback into ongoing improvements.
Regularly attend the Nashville, Tennessee office; live nearby or be willing to relocate.