Principal Site Reliability Engineer

Palo Alto Networks • Office - USA - CA - Headquarters, CA

2d • Onsite

About The Position

As a Principal Site Reliability Engineer within the Cortex DevOps team, you will serve as a technical leader driving reliability, scalability, observability, and operational excellence across the Cortex platform. You will partner with engineering, product, and infrastructure teams to influence architecture decisions, establish reliability standards, and build solutions that improve service availability and performance at global scale. The role calls for deep expertise in cloud infrastructure, observability, distributed systems, automation, and incident management. You will help shape the direction of observability and reliability platforms, mentor engineers, and promote best practices across the organization.

Requirements

• At least 10 years of experience in Site Reliability Engineering, DevOps, Cloud Engineering, or related disciplines.
• Deep expertise with Prometheus, Thanos, Grafana, OpenTelemetry, and modern observability platforms.
• Strong understanding of SRE principles, including SLIs, SLOs, error budgets, incident management, and operational excellence.
• Expert knowledge of Google Cloud Platform (GCP), Amazon Web Services (AWS), or similar cloud platforms.
• Expert-level experience with Kubernetes, Docker, and cloud-native architectures.
• Strong software engineering and automation skills using Python, Linux, Terraform, Ansible, and GitOps practices.
• Proven ability to influence technical direction and drive cross-functional initiatives across multiple engineering teams.

Nice To Haves

• Experience building and operating observability platforms at large scale.
• Experience implementing AI-driven operational tooling, automation, or AIOps solutions.
• Strong communication and leadership skills, with experience mentoring senior engineers and leading complex technical initiatives.
• Ability to operate independently, influence stakeholders, and drive outcomes across organizational boundaries.

Responsibilities

• Define and drive reliability, observability, and operational excellence standards across Cortex services and infrastructure.
• Design and evolve large-scale observability platforms using Prometheus, Thanos, Grafana, OpenTelemetry, and cloud-native monitoring solutions.
• Partner with engineering teams to ensure services are designed, instrumented, and operated with reliability and scalability in mind.
• Improve monitoring, alerting, incident management, and service health to proactively identify and prevent customer-impacting issues.
• Lead initiatives focused on automation, self-healing systems, operational efficiency, and reducing operational toil.
• Influence architectural decisions and technology adoption to improve platform reliability, performance, and cost efficiency.
• Mentor engineers and provide technical leadership across multiple teams and organizations.
• Evaluate emerging technologies and industry trends and implement solutions that advance Cortex's operational capabilities.
• Provide leadership during major incidents and drive post-incident reviews focused on systemic improvements.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed