Senior Site Reliability Engineer (SRE) (Hybrid)

Cisco•San Francisco, CA

1d•Hybrid

About The Position

The Splunk Agent Resilience team is defining the future of AI resilience. Our team provides scalable, cost-effective evaluation and guardrails that ensure AI agents behave as intended, improving reliability and reducing risks. This unified approach empowers our customers to confidently deploy and manage AI-powered applications with enhanced observability and control. As a Senior Site Reliability Engineer (SRE), you will build, operate, and continuously improve the reliability, scalability, and operational excellence of Splunk Agent Observability's deployment platform and production infrastructure. You will own the operational backbone supporting both cloud and air-gapped customer deployments, develop automation to improve operational efficiency, and partner closely with engineering teams to deliver reliable, secure, and scalable systems.

Requirements

7+ years’ experience with a Bachelor's degree or 4+ yrs with a Masters or 1 year with a PhD, or equivalent related experience; Experience to include at least 4 years in Site Reliability Engineering, Platform / Cloud / Infrastructure Engineering, or related fields
3+ years’ operating Kubernetes in production; experience with Helm
Experience building and maintaining CI/CD platforms and deployment automation
Experience working with AWS, GCP, or similar cloud platforms

Nice To Haves

Experience improving production reliability, scalability, and availability
Experience with monitoring, logging, observability, and alerting platforms
Strong scripting or programming skills in Python and/or Go
Experience with Infrastructure as Code tools such as Terraform
Experience with MLOps preferred
Solid understanding of networking fundamentals (VPCs, DNS, routing, load balancing)
Experience operating databases and tuning them for performance and reliability
Experience deploying and supporting cloud and air-gapped/on-prem environments
Strong debugging and troubleshooting skills across distributed systems

Responsibilities

Operate and improve Kubernetes-based production infrastructure and deployment systems
Own customer deployments across cloud and air-gapped environments, including installation, upgrades, troubleshooting, and lifecycle management
Build and improve deployment observability, monitoring, logging, and alerting
Improve reliability, scalability, and operational efficiency through automation and performance optimization
Participate in production incident response, root cause analysis, and reliability improvements
Tune infrastructure components—including databases and services—to improve performance and resilience
Design and develop internal tooling using Python and/or Go
Debug complex production issues spanning Kubernetes, networking, storage, and application layers
Manage infrastructure using Terraform or similar Infrastructure as Code tools
Collaborate with software engineers and customers to design secure, scalable, and reliable deployment architectures

Benefits

medical, dental and vision insurance
a 401(k) plan with a Cisco matching contribution
paid parental leave
short and long-term disability coverage
basic life insurance
10 paid holidays per full calendar year, plus 1 floating holiday for non-exempt employees
1 paid day off for employee’s birthday
paid year-end holiday shutdown
4 paid days off for personal wellness determined by Cisco
16 days of paid vacation time per full calendar year, accrued at rate of 4.92 hours per pay period for full-time employees (non-exempt)
flexible vacation time off program (exempt)
80 hours of sick time off provided on hire date and each January 1st thereafter
up to 80 hours of unused sick time carried forward from one calendar year to the next
Additional paid time away may be requested to deal with critical or emergency issues for family members
Optional 10 paid days per full calendar year to volunteer
annual bonuses (non-sales roles)
performance-based incentive pay (sales roles)
grants of Cisco restricted stock units