Senior Engineer - AI and HPC Observability

NVIDIA-posted 4 days ago

Full-time • Mid Level

Austin, CA

5,001-10,000 employees

Resume

Match Score

Upload and Match ResumeTrack Jobs with Teal

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Superclusters (MARS) builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you’ll help design solutions that power some of the world’s most advanced computing workloads. Observability is at the heart of this transformation. We are looking for a Senior AI & HPC Observability Engineer to design and build the next-generation observability platform for large-scale AI workloads, GPU clusters, and high-performance computing environments. This role blends deep technical engineering with large-scale data systems and developing scalable telemetry pipelines, AI-driven insights, and intelligent monitoring across NVIDIA’s world-class GPU infrastructure.

Design and implement full-stack observability systems covering metrics, logs, traces, and events for GPU-powered AI and HPC workloads.
Build large-scale telemetry data pipelines leveraging OpenTelemetry, Kafka, Prometheus, and other distributed systems to ingest, process, and analyze massive data streams.
Develop analytics and anomaly detection frameworks to enable real-time visibility, performance optimization, and predictive insights across multi-tenant environments.
Architect and tune high-throughput data stores (e.g., TSDBs, columnar databases, OLAP systems) for large-scale observability data.
Drive self-service analytics capabilities through APIs, dashboards, and recommendation engines that empower developers and operators with actionable insights.
Collaborate with AI platform, GPU, and cloud infrastructure teams to optimize observability for model training, inference workloads, and HPC performance.
Leverage machine learning and statistical techniques for correlation, anomaly detection, and intelligent alerting.
Contribute to performance tuning, scalability, and reliability of observability services across on-prem, and cloud environments.

BS or equivalent experience in Computer Science, Computer Engineering, or a related technical field.
8+ years of experience in large-scale observability, data engineering, or performance monitoring systems.
Proven expertise in building and scaling observability stacks (metrics, logs, traces, events) using OpenTelemetry, Prometheus, Grafana, or Thanos .
Deep understanding of data collection, transformation, and storage at scale, experience with streaming frameworks (Kafka, Flink, Spark) preferred.
Hands-on experience with Python, Go, and/or Java for backend development and automation.
Strong knowledge of API design, data modeling, SQL/NoSQL , and data pipeline architecture.
Experience working with PromQL , time-series databases, and large-scale monitoring systems.
Familiarity with AI/ML pipelines, GPU-based workloads , and HPC environments.
Experience with anomaly detection, log analytics, and recommendation systems using ML or statistical techniques.
Excellent problem-solving, debugging, and performance-tuning skills in distributed systems.

Proven experience designing and scaling full-stack observability platforms for large-scale AI, GPU, or HPC environments.
Hands-on expertise with OpenTelemetry , Prometheus , Kafka , and distributed data pipelines handling high-volume telemetry streams.
Strong background in data engineering, performance tuning, and time-series data modeling for real-time analytics.
Demonstrated use of machine learning or statistical techniques for anomaly detection, correlation, or intelligent alerting.
Deep understanding of API design , self-service observability, and building platforms that empower internal developers and operators.

You will also be eligible for equity and benefits .
Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

Track Jobs with Teal

Job Search Resources

•

AI Resume Builder

•

AI Engineer Resume Examples

•

AI Engineer Cover Letter Examples

Senior Engineer - AI and HPC Observability

Job Search Resources

Tools

Career Hubs

Guides

Company