Principal Software Engineer, AIOps and Observability

Nvidia•Santa Clara, CA

About The Position

NVIDIA has been reinventing computer graphics, PC gaming, and accelerated computing for 30 years. It is a unique legacy of innovation that's fueled by great technology and amazing people. Today, we're tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, generative AI , robots, and self-driving cars that can understand the world. Doing what's never been done before takes vision, innovation, and the world's best talent. As an NVIDIAN, you'll be immersed in a diverse, supportive environment where everyone is inspired to do their best work. We are looking for a highly skilled Principal Software Engineer to design and develop AIOps & Observability platforms at NVIDIA. The platforms are used by internal teams to monitor, diagnose, and optimize the products, millions of assets and services in cloud, on-prem, data centers, supply chain, and edge. You will work with a team of engineers, product managers, and partners to define the observability strategy, roadmap, and standard methodologies for NVIDIA. You will also mentor and coach other engineers on observability, machine learning, tools and techniques.

Requirements

Bachelor's degree in computer science and engineering, or related field, or equivalent experience.
15+ years of experience in product development and full stack engineering, with 5+ years of experience in developing and operating observability platforms and solutions, preferably in a cloud-native environment.
Strong knowledge and experience with observability tools, such as Prometheus, Victoria Metrics, Vector, Loki, Grafana, Alert Manager, Clickhouse, OpenTelemetry, etc.
Hands-on knowledge in AIOps tools such as BigPanda, PagerDuty, Datadog, etc.
Experience with Kubernetes, Nomad, Docker, and microservices architectures as well as experience with streaming services to ingest billions of events using NATS, Kafka, etc
Proficient in one or more programming languages, such as Go, Python, Java, C#, etc.
Passionate about observability and delivering high-quality internal platforms.
Experience with developing Observability solutions to monitor On-prem and Public Cloud environments.
Experience with running large Observability platforms on BareMetal Infrastructure
Establish scalable data pipelines and instrumentation for collecting, aggregating, and visualizing telemetry and operational metrics.

Nice To Haves

Deep understanding of implementing Observability solutions to large scale on-prem Infrastructure and Networking.
Hands-on experience with managing large scale Observability Platforms with LLMs & ML Models and building custom services to ingest billions of metrics and logs from wide range of assets.
Developed unified cloud observability platform to monitor Network, Compute, Power, Storage, Operating Systems, Security, Applications, SaaS Platforms.
Demonstrated experience and expertise in using machine learning and Generative AI to develop solutions such as predictive monitoring, incident diagnosis, summarization and correlation.
Demonstrate proficiency in AI/ML systems, generative AI, or agentic AI frameworks.

Responsibilities

Lead the design, development, and deployment of AIOps & Observability platforms, including metrics, logs, traces, events, alerts, dashboards, and visualizations.
Drive the technical vision and roadmap for AIOps and Observability initiatives, aligning with business goals and industry best practices.
Collaborate with other teams and customers to understand their observability needs and provide solutions that meet their requirements and expectations.
Establish and implement observability standards, guidelines, and processes across NVIDIA.
Research, evaluate, and adopt new observability technologies and frameworks that can enhance user experience.
Provide peer reviews to other engineers including feedback on performance, scalability, security and correctness.
Work with Data scientists to implement machine learning models for anomaly detection, forecasting, and root cause analysis on logs, metrics, and events.
Handle large volumes of data and ensure data quality, security, and compliance.
Develop and operate scalable, reliable, and distributed systems that can handle high traffic and complex workloads.
Find opportunities to automate remediation of commonly occurring issues to operate systems reliably and efficiently.