Hardware Health

Microsoft AI•Mountain View, NY
•Hybrid

About The Position

Microsoft AI operates advanced AI training infrastructure, including multi-gigawatt clusters with high-performance GPUs, ultra-low-latency NVLink/NVSwitch networks, and liquid-cooling systems. The Member of Technical Staff - Hardware Health helps ensure sustained reliability, performance, and availability across large-scale deployments. The role works with research, hardware, datacenter, and platform engineering teams to develop predictive health models, failure detection frameworks, and autonomous remediation systems for AI clusters. Microsoft AI develops Copilot and other consumer AI products and research, including Copilot, Bing, Edge, and generative AI research. Employees are expected to work from a designated Microsoft office at least four days per week if they live within 50 miles of that office in the U.S. or 25 miles outside the U.S., subject to local law and jurisdiction-specific variation.

Requirements

  • Bachelor's degree in Computer Science or a related technical field and 6+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • Experience working with large-scale HPC or GPU systems, such as NVIDIA H100/GB200 or equivalent.
  • Deep understanding of GPU architecture, high-speed interconnects such as NVLink, InfiniBand, and RoCE, and large datacenter topologies.
  • Proficiency with hardware telemetry, diagnostics, or failure-analysis tools.
  • Experience with exascale-class systems or cloud-scale AI clusters.
  • Familiarity with reliability modeling, machine-learning-based anomaly detection, or predictive maintenance.

Nice To Haves

  • Master's degree in Computer Science or a related technical field and 8+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • Bachelor's degree in Computer Science or a related technical field and 12+ years of technical engineering experience coding in languages such as C, C++, C#, Java, JavaScript, or Python, or equivalent experience.
  • Contributions to large-scale infrastructure operations, supercomputing centers, or AI hardware design.

Responsibilities

  • Design and develop hardware health monitoring and diagnostic frameworks for large GPU clusters, including NVL16/NVL72/GB200+ scale.
  • Build predictive analytics pipelines using telemetry, power, and thermal data to anticipate hardware degradation and systemic issues.
  • Collaborate with silicon, firmware, and datacenter engineers to identify root causes and remediate large-scale hardware anomalies.
  • Define system health KPIs, including NIS/RIS, MTBF, and failure domain analysis, and integrate them into real-time observability platforms.
  • Lead incident triage for high-impact GPU, network, and cooling issues across distributed clusters.
  • Drive health-management automation to reduce manual intervention to the top 5% of anomalies.
  • Partner with cross-functional teams to influence hardware design for reliability, thermal efficiency, and serviceability.
  • Embody Microsoft’s Culture and Values.

Benefits

  • Certain roles may be eligible for benefits and other compensation.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service