Principal Software Engineer - Scaled Manufacturing, Core Infrastructure

Oracle•Santa Clara, CA
•$114,600 - $234,600

About The Position

Scaled Manufacturing (SMF) builds and operates container-based services that validate infrastructure across factory environments. It supports repeatable production of liquid-cooled GPU platforms, including GB200, GB300, VR, MI355, and MI455, by integrating quality, yield, throughput, and readiness controls into manufacturing. SMF provides factory infrastructure, a Control Plane for test lifecycle management and related workflows, a Data Plane for automated test validation, and manufacturing repair and triage capabilities to identify root causes and reduce repeat failures. The team works with Supply Chain Operations, Oracle Hardware Development, manufacturing partners, data-center teams, NVIDIA, and AMD to detect hardware, firmware, software, and configuration issues before racks ship. In this role, you will lead the development and architecture of scalable, fault-tolerant distributed systems for manufacturing validation and automation. You will optimize performance, reliability, and scalability; develop telemetry, monitoring, and automated testing; troubleshoot complex production issues; and drive infrastructure automation, security, operational readiness, and continuous improvement. You will also collaborate across engineering teams and mentor peers. The work includes developing GPU server provisioning and validation workflows, building burn-in and qualification tests, investigating failures across hardware and software layers, and using manufacturing quality and yield data to improve test coverage and deployment readiness. The role also covers infrastructure and networking engineering, host provisioning, containers, monitoring and observability, and validation tooling for large factory fleets.

Requirements

  • Experience bringing up, validating, or troubleshooting GPU, accelerator, or server hardware in a lab, manufacturing, or data-center environment.
  • Strong Linux skills, including operating system installation and imaging, kernel versions and modules, device drivers, boot processes, system logs, and hardware troubleshooting.
  • Ability to diagnose failures across hardware, firmware, operating system, kernel, and driver layers.
  • Knowledge of CPU and GPU architecture, PCIe, device enumeration, BIOS/UEFI, BMC interfaces, and power and thermal behavior.
  • Networking knowledge, including Linux network configuration, Smart NICs, IP addressing, DHCP/DNS, and troubleshooting network connectivity.
  • Experience scripting and automating validation or provisioning workflows using Python and shell.
  • Ability to develop dependable test workflows for large-scale systems, analyze results across many servers, and identify recurring failures or performance outliers.
  • Experience documenting test plans, failure investigations, and results while working across engineering and manufacturing teams.
  • Strong experience in data pipelines, data modeling, analytics and measurement systems, and applied AI pipelines.
  • Knowledge of high-performance networking concepts such as RDMA or RoCE.
  • Experience with network engineering and data-center/site networking, including VLANs, VxLAN firewalls, L3 networking, gNMI, VPNs, DHCP, and switch configuration.

Nice To Haves

  • Hands-on experience with both NVIDIA CUDA/DCGM and AMD ROCm/RVS environments.
  • Experience validating multi-GPU systems, GPU interconnects, PCIe bandwidth, and single-node or multi-node workloads.
  • Familiarity with PXE/iPXE provisioning, Redfish, PLDM, virtual media, firmware lifecycle management, and factory test infrastructure.
  • Experience with component traceability, manufacturing yield analysis, and corrective-action processes.
  • Ability to work on site at ODM facilities during platform bring-up and production ramps.
  • Experience with Terraform, containers, Prometheus, Grafana, and monitoring/observability.
  • Experience with identity provider tools such as Keycloak and FreeIPA.
  • Experience with Arista switch configuration.

Responsibilities

  • Lead the development and architecture of scalable, fault-tolerant distributed systems supporting manufacturing validation and automation.
  • Optimize system performance, reliability, and scalability; develop telemetry, monitoring, and automated testing; and troubleshoot complex production issues.
  • Drive infrastructure automation, security, operational readiness, and continuous improvement, while collaborating across engineering teams and mentoring peers.
  • Develop automated workflows for GPU server provisioning, configuration, firmware installation, validation, upgrades, and recovery.
  • Build and maintain GPU burn-in, qualification, and factory acceptance tests; define pass/fail criteria and document results for approved platform configurations.
  • Validate GPU health, device enumeration, memory, PCIe and GPU interconnects, networking, power, cooling, and performance under sustained workloads.
  • Use platform-specific diagnostics, including NVIDIA DCGM and CUDA-based tools, AMD ROCm Validation Suite (RVS), and AMD SMI.
  • Investigate failures across GPUs, server hardware, BIOS/UEFI, BMC, firmware, Linux kernels, drivers, operating systems, storage, and networking.
  • Capture logs, test results, firmware and software versions, and component identifiers to support traceability and root cause analysis.
  • Work with ODM engineers and internal and external partners to verify corrective actions and prevent repeat failures.
  • Use manufacturing yield, test throughput, failure trends, and quality metrics to improve test coverage and deployment readiness.
  • Develop reliable validation tooling that runs across large factory fleets and reports system health and test outcomes.
  • Partner with provisioning, platform, and data-center teams to ensure factory tests reflect deployment requirements.
  • Document test procedures, known issues, recovery steps, and production release recommendations.
  • Contribute to network engineering, data-center/site networking, network automation and troubleshooting, and host provisioning/configuration involving Terraform, containers, and monitoring/observability systems.

Benefits

  • Medical, dental, and vision insurance, including expert medical opinion.
  • Short-term disability and long-term disability.
  • Life insurance and AD&D; supplemental life insurance for employees, spouses, and children.
  • Health care and dependent care Flexible Spending Accounts.
  • Pre-tax commuter and parking benefits.
  • 401(k) Savings and Investment Plan with company match.
  • Paid time off, including flexible vacation for eligible salaried non-overtime positions or accrued vacation for other eligible employees.
  • 11 paid holidays.
  • Paid sick leave: 72 hours upon hire, refreshed annually, with unused time carrying over up to a 112-hour maximum cap.
  • Paid parental leave.
  • Adoption assistance.
  • Employee Stock Purchase Plan.
  • Financial planning and group legal benefits.
  • Voluntary benefits including auto, homeowner, and pet insurance.
  • The role may be eligible for bonus, equity, and compensation deferral.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service