Staff Site Reliability Engineer
About The Position
The Splunk Agent Resilience team develops scalable, cost-effective evaluation and guardrails to help ensure AI agents behave as intended, improving reliability and reducing risk. The team supports customers deploying and managing AI-powered applications through enhanced observability and control. The Staff Site Reliability Engineer provides technical leadership for the reliability, scalability, and operational architecture of Splunk Agent Observability's platform. The role defines long-term reliability strategy, leads major infrastructure initiatives, and drives engineering excellence across deployment automation, production operations, and platform resiliency. It owns complex production systems, influences engineering direction across teams, and establishes best practices for large-scale cloud and on-premises deployments. This is a hybrid position requiring on-site work Tuesdays and Thursdays. In California, Tuesdays are on-site in San Francisco and Thursdays in San Jose; in New York City, both days are on-site in the NYC office.
Requirements
- At least 8 years of experience with a bachelor's degree, 6 years with a master's degree, or 3 years with a Ph.D., or equivalent related experience; experience must include at least 6 years in Site Reliability Engineering, Platform, Cloud, Infrastructure Engineering, or related fields.
- At least 5 years operating large-scale Kubernetes platforms in production.
- Experience designing highly available, scalable, and resilient distributed systems.
- Experience with AWS, GCP, or other public cloud platforms.
- Strong experience designing CI/CD platforms and deployment automation at scale.
Nice To Haves
- Expertise in observability, monitoring, alerting, capacity planning, and performance engineering.
- Strong programming skills in Python and/or Go.
- Deep experience with Infrastructure as Code, such as Terraform.
- Experience with MLOps.
- Strong understanding of networking, distributed systems, storage, databases, and cloud architecture.
- Experience operating both SaaS and enterprise/on-premises deployments.
- Demonstrated technical leadership across multiple engineering teams.
- Experience leading incident management, postmortems, and long-term reliability initiatives.
Responsibilities
- Define and drive the technical roadmap for platform reliability, scalability, and operational excellence.
- Lead the architecture and evolution of deployment platforms supporting cloud and air-gapped customer environments.
- Establish reliability engineering standards, including service level objectives (SLOs), operational readiness, capacity planning, and resiliency reviews.
- Lead major reliability and scalability initiatives across Kubernetes, deployment infrastructure, databases, and networking.
- Drive automation to eliminate operational toil and improve engineering productivity.
- Design and build internal platforms, frameworks, and tooling for reliable operations at scale.
- Lead complex production incident response and drive systemic improvements through root cause analysis and long-term remediation.
- Partner with engineering leadership to influence platform architecture, deployment strategy, and production readiness.
- Mentor engineers and raise engineering standards through technical leadership, design reviews, and operational guidelines.
- Collaborate with customers and internal teams to build secure, scalable, and highly reliable deployment architectures for cloud and on-premises environments.
Benefits
- Medical, dental, and vision insurance, subject to plan eligibility rules.
- 401(k) plan with Cisco matching contribution.
- Paid parental leave.
- Short- and long-term disability coverage.
- Basic life insurance.
- Cisco restricted stock unit grants may be available, subject to continued-employment vesting requirements.
- 10 paid holidays per full calendar year, plus one floating holiday for non-exempt employees.
- One paid day off for an employee’s birthday, a paid year-end holiday shutdown, and four paid personal wellness days.
- Non-exempt employees receive 16 days of paid vacation per full calendar year, accrued at 4.92 hours per pay period for full-time employees.
- Exempt employees participate in a flexible vacation time off program with no defined usage limit, subject to availability and business limitations.
- 80 hours of sick time off provided on hire and each January 1, with up to 80 unused hours carried forward.
- Additional paid time away may be requested for critical or emergency family issues.
- Optional 10 paid volunteer days per full calendar year.
- Employees in non-sales roles are eligible to earn annual bonuses, subject to Cisco policies.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
