Principal Core Infrastructure Engineer - GPU Validation
About The Position
Join Oracle’s HPC team to develop automated tools and validation frameworks that ensure GPU racks are ready for production. The role focuses on GPU infrastructure validation, including NCCL performance benchmarks, GPU health and NVLink connectivity tests, Slurm-based validation workflows, and automated FDT and cable diagnostics. It also designs, implements, and optimizes distributed-system components with emphasis on scalability, resiliency, and operability. The engineer delivers features and performance tests, builds fault-tolerant paths, implements recovery mechanisms, and uses tests, telemetry, dashboards, and alerts to detect and mitigate issues. The position develops automation and infrastructure-as-code tools, supports incident response and root-cause investigations, and applies security, change-management, compliance, and documentation practices. The employee must live or be willing to relocate to regularly work from Oracle’s Nashville, Tennessee office.
Requirements
- Must live or be willing to relocate to regularly work from the Nashville, TN office.
- The posting does not specify a required degree or a minimum number of years of experience.
Responsibilities
- Design and develop scalable validation frameworks for GPU racks and HPC infrastructure.
- Build automated benchmarks and diagnostics for NCCL performance, GPU health, NVLink connectivity, InfiniBand, and high-speed networks.
- Develop Slurm-based workflows for distributed validation and performance testing.
- Implement retries, timeouts, redundancy, replication, automatic failover, and other recovery mechanisms to improve resilience.
- Develop functional, load, performance, and fault-injection tests to assess reliability and production readiness.
- Build dashboards, telemetry, and alerts to monitor infrastructure health and validation results.
- Develop automated Linux diagnostics, log analysis, hardware fault isolation, FDT, and cable-validation tools.
- Convert incident findings into reusable diagnostics and preventive automation.
- Develop recovery runbooks and contribute to incident response and root-cause investigations.
- Develop and maintain Bash, Python, and Infrastructure as Code tools for deployment, configuration, troubleshooting, and maintenance.
- Implement access controls and secure handling of infrastructure credentials and diagnostic data.
- Automate patching, updates, and rollback workflows in accordance with change-management practices.
- Maintain technical documentation and improve automation coverage across infrastructure workflows.
- Translate project requirements into development plans and deliverables; manage assigned work with minimal supervision and meet agreed timelines and quality standards.
- Adapt priorities as requirements, resources, and dependencies change.
- Collaborate with HPC, infrastructure, and engineering teams to define requirements and align on outcomes.
- Investigate issues using logs, metrics, and diagnostic results; resolve issues independently, escalate when needed, and document findings.
- Build expertise in GPU systems, HPC technologies, and software development through training and feedback.
- Recommend and implement improvements to validation coverage, reliability, and development efficiency; share reusable tools and best practices and incorporate team feedback.
- Review peers’ implementations for scalability compliance.
Benefits
- Medical, dental, and vision insurance, including expert medical opinion.
- Short-term and long-term disability insurance.
- Life insurance and AD&D; supplemental life insurance for employees, spouses, and children.
- Health care and dependent care Flexible Spending Accounts.
- Pre-tax commuter and parking benefits.
- 401(k) Savings and Investment Plan with company match.
- Paid time off, including flexible or accrued vacation depending on eligibility and position.
- 11 paid holidays.
- Paid sick leave.
- Paid parental leave.
- Adoption assistance.
- Employee Stock Purchase Plan.
- Financial planning and group legal benefits.
- Voluntary auto, homeowner, and pet insurance.
- May be eligible for bonus, equity, and compensation deferral.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
Education Level
No Education Listed
