AI Observability Engineering Technical Leader
About The Position
The Platform and Security team within Cisco’s AI Software and Platform Group designs and operates infrastructure for Cisco’s AI-powered offerings, including Cisco Cloud Control, AI Defense, and AI Canvas. The Observability Engineering group provides the metrics, traces, and logs needed to keep these services observable and ensure important workflows are continuously monitored. This role partners across the AI Software and Platform group to set observability standards, deliver platform solutions, and improve production reliability. The role defines the observability platform vision and executes it end-to-end in partnership with software teams. It drives distributed tracing coverage, OpenTelemetry enablement, logging standards, and cardinality management, while operating Splunk and telemetry pipeline components at scale. It also advances an agentic observability outcome in which agents analyze health signals and provide distilled, actionable information more rapidly than conventional monitoring.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field and 12+ years of experience in a platform services or DevOps role with specific observability ownership; or a Master’s degree and 8+ years of related experience; or a Ph.D. and 5+ years of related experience.
- At least 2 years of experience with Splunk Cloud and Splunk Observability or SignalFx, using Splunk Cloud for logging and Splunk Observability for metrics and traces.
- Experience with distributed tracing concepts and implementation, including instrumentation approaches and strategies for managing scope and overhead.
- Experience coding in Python and/or Go, including building libraries, automation, or operational tooling for observability platforms.
- Experience with distributed systems infrastructure, including Kubernetes, OpenTelemetry collectors, and infrastructure-as-code such as Terraform.
- Familiarity with Helm charts and modern SDLC practices such as GitHub workflows.
Nice To Haves
- Experience driving large-scale distributed tracing initiatives across an organization with 100 or more microservices.
- Experience using AI agents to continually refine observability outcomes, such as detecting issues, improving routing or alerts, and recommending instrumentation changes.
- Understanding of LLM observability, including familiarity with major providers and patterns for measuring reliability, latency, and quality signals.
- Deep understanding of AWS services commonly used in telemetry pipelines and distributed architectures.
Responsibilities
- Define the observability platform vision and execute it end-to-end in partnership with software teams.
- Partner with development teams to instrument services correctly and consistently.
- Deliver 100% distributed tracing coverage, balancing instrumentation scope and overhead.
- Own and evolve OpenTelemetry enablement, including libraries, process documents, and design patterns for Go and Python applications.
- Drive observability best practices for logging standards and cardinality management.
- Operate and maintain Splunk Platform, Splunk Observability, OpenTelemetry collectors, and telemetry pipelines for dependable operation at scale.
- Advance an agentic observability outcome in which agents analyze observability health signals and deliver distilled, actionable information.
Benefits
- Medical insurance
- Dental insurance
- Vision insurance
- 401(k) plan with Cisco matching contribution
- Paid parental leave
- Short- and long-term disability coverage
- Basic life insurance
- Paid holidays and a floating holiday for non-exempt employees
- Birthday day off
- Paid year-end holiday shutdown
- Paid personal wellness days
- Paid vacation or flexible vacation time off, depending on employee classification
- Paid sick time
- Additional paid time away may be requested for critical or emergency family issues
- Optional paid volunteer days
- Annual bonus eligibility for non-sales roles, subject to Cisco policies
- Potential eligibility for Cisco restricted stock unit grants
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
