Principal Software Development Engineer- Go

Zscaler•Santa Clara, CA

1d•$185,000 - $265,000•Hybrid

About The Position

Zscaler is seeking a Principal Software Development Engineer to build the Service Platform & Orchestration capabilities for the Zero Trust Exchange department. This hybrid role requires working in the office three days per week and reports to the Director, Software Engineering. The role focuses on engineering the management plane and reliability systems that govern the lifecycle of ZIA’s global fleet. It is a hands-on position leading the migration from legacy automation to a stateful, durable management plane built on Temporal, enabling deterministic provisioning and lifecycle operations. The engineer will treat infrastructure as a distributed system and develop self-healing capabilities and AI-driven SRE practices for a fleet of more than 100,000 instances. The position involves building replay-safe, idempotent workflows; evolving the global fleet into an eventually consistent distributed system; developing LLM- and ML-enabled services for signal correlation, triage, and self-correcting operations; creating framework-level services and internal APIs; and embedding telemetry that makes fleet-wide actions explainable, auditable, and replayable. Candidates should be comfortable with ambiguity, act with ownership and urgency, connect their work to global-scale outcomes, and remain adaptable under pressure. Zscaler describes its benefits as including various health plans, vacation and sick time, parental leave options, retirement options, education reimbursement, and in-office perks. The base salary range for this full-time position is $185,000–$265,000 USD, excluding commission, bonus, or equity where applicable.

Requirements

Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes in the functional domain.
BS/MS in Computer Science or a related technical field and 12+ years of experience in hyperscale systems.
Deep understanding of failure modes and technical challenges that emerge at massive scale.
Mastery of backend systems languages, with a strong preference for Go; alternatively, deep expertise in Java, C++, or Python and demonstrated ability to rapidly adopt Go.
Track record of setting high standards for code quality and distributed-system correctness.
Strong experience designing and operating complex distributed systems, particularly addressing concurrency, failure handling, and performance optimization.
Proven experience developing Platform APIs (REST/gRPC) with strong guarantees for idempotency, verification, and safe rollout patterns.

Nice To Haves

Proficiency with AI code-assistance tools such as Cursor or Windsurf to accelerate legacy refactoring and system development.
Proficiency in PostgreSQL or other relational stores used for high-scale, stateful management-plane services.
Direct experience building or operating systems with Temporal.io, Cadence, or similar workflow engines.

Responsibilities

Lead hands-on development and migration to a workflow-as-code platform using Temporal, building replay-safe, idempotent workflows for deterministic operations at global scale.
Evolve scripted automation into a robust management plane that treats the global fleet as a single, eventually consistent distributed system.
Design and implement services using LLMs and ML for intelligent signal correlation, automated triage, and self-correcting fleet operations.
Develop framework-level services and internal APIs so new products are orchestration-ready, with reliability hooks built into the code.
Build metrics, traces, and events into the management plane so fleet-wide actions are explainable, auditable, and replayable.