AI Observability Engineering Technical Leader

Cisco•Milpitas, MI

About The Position

The Platform and Security team within Cisco’s AI Software and Platform Group designs and operates the infrastructure for Cisco’s AI-powered offerings, including Cisco Cloud Control, AI Defense, and AI Canvas. The Observability Engineering group provides the metrics, traces, and logs needed to keep these services observable and to monitor important workflows. This role partners across the AI Software and Platform group to set observability standards, deliver platform solutions, and improve production reliability. It also advances agentic observability so health insights can be surfaced more quickly and intelligently than through traditional dashboards and alerts. The role defines the observability platform vision and executes it end-to-end in partnership with software teams. It owns and evolves OpenTelemetry enablement for Go and Python applications, establishes logging and cardinality-management practices, and operates Splunk, OpenTelemetry collectors, and telemetry pipelines at scale. It also works toward enabling agents to analyze observability health signals and provide distilled, actionable information.

Requirements

Bachelor’s degree in Computer Science, Engineering, or a related field and 12+ years of experience in a platform services or DevOps role with specific observability ownership; alternatively, a master’s degree and 8+ years of related experience, or a Ph.D. and 5+ years of related experience.
2+ years of experience with Splunk Cloud and Splunk Observability or SignalFx, using Splunk Cloud for logging and Splunk Observability for metrics and traces.
Experience with distributed tracing concepts and implementation, including instrumentation approaches and strategies for managing scope and overhead.
Experience coding in Python and/or Go, including building libraries, automation, or operational tooling that supports observability platforms.
Experience with distributed-systems infrastructure, including Kubernetes, OpenTelemetry collectors, and infrastructure-as-code such as Terraform.
Familiarity with Helm charts and modern SDLC practices such as GitHub workflows.

Nice To Haves

Experience driving large-scale distributed-tracing initiatives across 100+ microservices.
Experience using AI agents to continually refine observability outcomes, such as detecting issues, improving routing or alerts, and recommending instrumentation changes.
Understanding of LLM observability, including familiarity with major providers and patterns for measuring reliability, latency, and quality signals.
Deep understanding of AWS services commonly used in telemetry pipelines and distributed architectures.

Responsibilities

Define the observability platform vision and execute it end-to-end.
Partner with software teams to instrument services correctly and consistently.
Deliver 100% distributed tracing coverage in collaboration with development teams, balancing instrumentation value against scope and overhead.
Own and evolve OpenTelemetry enablement, including libraries, process documents, and design patterns that accelerate adoption in Go and Python applications.
Drive observability best practices for logging standards and cardinality management.
Operate Splunk Platform, Splunk Observability, OpenTelemetry collectors, and telemetry pipelines to keep them dependable at scale.
Advance agentic observability so agents can analyze health signals and deliver distilled, actionable information more rapidly than conventional monitoring.

Benefits

Medical, dental, and vision insurance, subject to plan eligibility rules.
401(k) plan with a Cisco matching contribution.
Paid parental leave.
Short- and long-term disability coverage.
Basic life insurance.
Cisco restricted stock unit grants may be available, subject to eligibility and continued-employment vesting conditions.
10 paid holidays per full calendar year.
One floating holiday for non-exempt employees.
One paid day off for an employee’s birthday.
Paid year-end holiday shutdown.
Four paid days off for personal wellness.
Non-exempt employees receive 16 days of paid vacation per full calendar year, accrued at 4.92 hours per pay period for full-time employees.
Exempt employees participate in a flexible vacation time-off program with no defined limit, subject to availability and business limitations.
80 hours of sick time provided on the hire date and each January 1, with up to 80 hours of unused sick time carried forward.
Additional paid time away may be requested for critical or emergency family-member issues.
Optional 10 paid volunteer days per full calendar year.
Employees in non-sales roles are eligible to earn annual bonuses, subject to Cisco policies.