Sr. Production Engineer

Zscaler•San Jose, CA

2d•Hybrid

About The Position

Zscaler is hiring a Sr. Production Engineer in its Cloud Infrastructure & Operations department. The role is hybrid, with three days per week in San Jose, California, or Bellevue, Washington, and reports to the Senior Manager, Site Reliability Engineering. The engineer will help ensure the reliability of Zscaler’s global security platform by providing technical vision and hands-on execution to promote an automation-first culture. The work includes maturing observability and architectural standards to reduce Mean Time to Mitigate and support the scalability of globally distributed, multi-cloud infrastructure. The position involves designing and implementing highly available infrastructure across public cloud and bare-metal environments, automating operational work, establishing observability and reliability practices, leading incident response, and partnering with engineering teams on operability. The engineer will participate in a 24/7 on-call rotation and use incident data and ITIL practices to improve service maturity.

Requirements

Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes in a functional domain.
At least 4 years of experience managing reliability, scalability, and availability for large-scale production services.
Deep programming expertise, for example in Python, Go, or C/C++.
Strong background in networking protocols, Linux/FreeBSD systems, and distributed architecture.
Experience in high-stakes incident management and participation in a 24/7 on-call rotation.
Proficiency in ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews.

Nice To Haves

Ability to leverage AI technologies and workflows to drive measurable operational efficiency.
Experience implementing AI-driven AIOps tools, machine-learning models for anomaly detection, or predictive capacity planning to improve infrastructure reliability and automate incident response.
Extensive experience with public cloud (AWS, Azure, GCP), Infrastructure-as-Code (Ansible, Terraform), and large-scale chaos engineering or disaster recovery planning.
Expertise in global routing (BGP), traffic tunneling (GRE, IPSec), L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internals.

Responsibilities

Design and implement highly available, scalable infrastructure across AWS, Azure, GCP, and bare-metal environments.
Promote an automation-first culture by writing Python or Go code to eliminate manual toil and build self-healing systems.
Implement and maintain observability using Prometheus, Grafana, and OpenTelemetry; define SLIs and SLOs and establish error budgets.
Serve as a lead Incident Commander on the TDO on-call rotation, develop response playbooks, and conduct in-depth post-incident analyses.
Partner with Engineering and other teams to conduct operability reviews.
Use incident data and systematic problem management to drive service maturity.