Principal Software Development Engineer (Microservices)

Zscaler•Santa Clara, CA

8h•Hybrid

About The Position

Zscaler is looking for a Principal Software Development Engineer (Microservices) to join their team. This is a hybrid role based in Santa Clara, CA, requiring a minimum of 3 days per week in the office. The role reports to the Senior Director, Software Development Engineering in the Engineering ZIA Core department. The engineer will work on the control plane for the Zscaler Internet Access (ZIA) product, focusing on evolving the platform to handle massive scale using a cloud-native, microservices-based architecture. The company emphasizes a future of work that is Human + AI, aiming to amplify human potential with machine intelligence to solve complex security challenges. Zscaler's core behaviors include ownership and collaboration, trust through outcomes and impact, and a challenge culture with ongoing feedback.

Requirements

Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain.
12+ years in software development with 4+ years building and operating production microservices at significant scale, including proficiency in Go.
Deep experience with distributed systems and messaging (Kafka preferred; SQS/RabbitMQ acceptable), alongside patterns such as CQRS and Event Sourcing where fit-for-purpose.
Kubernetes in AWS at scale (EKS preferred), strong Docker fundamentals, Infrastructure-as-Code, and GitOps workflows.
Strong SQL (PostgreSQL/MySQL/Amazon RDS/Aurora) and NoSQL (Cassandra/MongoDB/DynamoDB) design and scaling with familiarity with multi-region data trade-offs, alongside performance engineering and cost optimization in AWS.
Comprehensive reliability and security experience including SLIs/SLOs, error budgets, metrics (Prometheus), tracing (OpenTelemetry), SLO-based alerting, incident management, postmortems, service mesh (Istio/Linkerd), mTLS, OAuth2/OIDC, and secrets management (KMS/Vault).

Nice To Haves

Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency.
Chaos engineering, advanced load/performance testing, fault injection, and disaster recovery drills combined with multi-region design expertise (active-active, partitioning, replication, eventual consistency) and migration leadership (strangler patterns, domain decomposition).
Event contract governance (Avro/Protobuf with schema registry), DLQ processing strategies, and data replay tooling.
OSS contributions, technical talks, patents, or published design writing.

Responsibilities

Lead the technical roadmap to modernize the ZIA control plane using API-first, domain-aligned microservices with clear service ownership and multi-region architectures (active-active, failover, disaster recovery).
Design, develop, and operate low-latency, event-driven services end to end (API through storage) utilizing resilience patterns, progressive delivery (blue/green, canary, feature flags), GitOps workflows (Argo CD/Flux), and safe rollout/rollback strategies.
Drive engineering excellence through standards, code reviews, testing, paved-road patterns, API and event contract governance (versioning, consumer-driven contracts, schema registry, DLQs), and cross-functional partnerships across Product, SRE, Security, Data, and Architecture while representing designs in customer/partner forums.
Own service reliability and operability end to end, including SLI/SLOs (p95/p99 latency, availability), SLO-based alerting, error budgets, incident response, postmortems, and advanced observability with distributed tracing (OpenTelemetry, Jaeger/Tempo) and structured logging.
Embed microservices security (OAuth2/OIDC, mTLS, service-to-service authZ, KMS/Vault), supply-chain integrity (SBOM, SLSA), performance optimization, capacity planning, and cloud spend/autoscaling across AWS primitives.