Senior Software Engineer, AIOps and Observability

NVIDIA•Santa Clara, CA

7d•$200,000 - $322,000

About The Position

NVIDIA is seeking a Senior Software Engineer to design, develop, and operate AIOps and observability platforms used by internal teams to monitor, diagnose, and optimize products, assets, and services across cloud, on-premises, data center, supply chain, and edge environments. The engineer will collaborate with engineers, product managers, partners, teams, and customers to shape the observability strategy, roadmap, standards, and methodologies, and will mentor and coach engineers in observability, machine learning, tools, and techniques. The role includes leading platform development across metrics, logs, traces, events, alerts, dashboards, and visualizations; building scalable, reliable distributed systems and data pipelines; and applying machine learning to anomaly detection, forecasting, and root cause analysis. The engineer will also develop AI agents and AI-native observability tools and workflows that reason across telemetry and incident history to help engineers detect, understand, and resolve production issues, including through automated debugging and remediation recommendations. The position requires experience with large-scale observability across on-premises, public cloud, and bare-metal infrastructure, as well as attention to data quality, security, and compliance. The expected base salary range is USD 200,000–322,000, determined by location, experience, and other objective job-related factors. The employee will also be eligible for equity and benefits. Applications will be accepted at least until November 3, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is an equal opportunity employer.

Requirements

Bachelor’s degree in computer science and engineering or a related field, or equivalent experience.
12+ years of experience in product development and full-stack engineering.
5+ years of experience developing and operating observability platforms and solutions, preferably in a cloud-native environment.
Strong knowledge and experience with observability tools such as Prometheus, VictoriaMetrics, Vector, Loki, Grafana, Alert Manager, ClickHouse, and OpenTelemetry.
Hands-on knowledge of AIOps tools such as BigPanda, PagerDuty, and Datadog.
Experience with Kubernetes, Nomad, Docker, and microservices architectures.
Experience with streaming services such as NATS and Kafka for ingesting billions of events.
Proficiency in one or more programming languages such as Go, Python, Java, or C#.
Experience developing observability solutions for on-premises and public cloud environments.
Experience running large observability platforms on bare-metal infrastructure.
Passion for observability and delivering high-quality internal platforms.

Nice To Haves

Deep understanding of implementing observability solutions for large-scale on-premises infrastructure and networking.
Hands-on experience managing large-scale observability platforms with LLMs and machine learning models, and building custom services to ingest billions of metrics and logs from a wide range of assets.
Experience developing a unified cloud observability platform to monitor network, compute, power, storage, operating systems, security, applications, and SaaS platforms.
Demonstrated experience using machine learning and generative AI to develop predictive monitoring, incident diagnosis, summarization, and correlation solutions.
Proficiency in AI/ML systems, generative AI, or agentic AI frameworks.

Responsibilities

Lead the design, development, and deployment of AIOps and observability platforms, including metrics, logs, traces, events, alerts, dashboards, and visualizations.
Drive the technical vision and roadmap for AIOps and observability initiatives, aligning them with business goals and industry best practices.
Collaborate with teams and customers to understand observability needs and deliver solutions that meet requirements and expectations.
Establish and implement observability standards, guidelines, and processes across NVIDIA.
Research, evaluate, and adopt observability technologies and frameworks to enhance user experience.
Review other engineers’ work and provide feedback on performance, scalability, security, and correctness.
Work with data scientists to implement machine learning models for anomaly detection, forecasting, and root cause analysis using logs, metrics, and events.
Handle large volumes of data while ensuring data quality, security, and compliance.
Develop and operate scalable, reliable distributed systems capable of handling high traffic and complex workloads.
Develop AI agents and AI-native observability tools to help engineers detect, understand, and resolve production issues faster.
Build agentic workflows that reason across logs, metrics, traces, events, alerts, topology, and incident history to support anomaly detection, forecasting, root cause analysis, automated debugging, and remediation recommendations.
Establish scalable data pipelines and instrumentation for collecting, aggregating, and visualizing telemetry and operational metrics.
Mentor and coach engineers on observability, machine learning, tools, and techniques.