Principal Software Development Engineer (Microservices)

Zscaler•Santa Clara, CA

20h•$185,500 - $265,000•Hybrid

About The Position

We are looking for a Principal Software Development Engineer (Microservices) to join our team. This is a hybrid role (in Santa Clara, CA office, minimum 3 days per week), reporting to the Senior Director, Software Development Engineering in the Engineering ZIA Core department. You will work on the control plane for the Zscaler Internet Access (ZIA) product, helping evolve the platform to meet massive scale with cloud-native, microservices-based architecture.

Requirements

Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI-driven solutions to optimize outcomes within your functional domain
12+ years in software development with 4+ years building and operating production microservices at significant scale, including proficiency in Go
Deep experience with distributed systems and messaging (Kafka preferred; SQS/RabbitMQ acceptable), alongside patterns such as CQRS and Event Sourcing where fit-for-purpose
Kubernetes in AWS at scale (EKS preferred), strong Docker fundamentals, Infrastructure-as-Code, and GitOps workflows
Strong SQL (PostgreSQL/MySQL/Amazon RDS/Aurora) and NoSQL (Cassandra/MongoDB/DynamoDB) design and scaling with familiarity with multi-region data trade-offs, alongside performance engineering and cost optimization in AWS
Comprehensive reliability and security experience including SLIs/SLOs, error budgets, metrics (Prometheus), tracing (OpenTelemetry), SLO-based alerting, incident management, postmortems, service mesh (Istio/Linkerd), mTLS, OAuth2/OIDC, and secrets management (KMS/Vault)

Nice To Haves

Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
Chaos engineering, advanced load/performance testing, fault injection, and disaster recovery drills combined with multi-region design expertise (active-active, partitioning, replication, eventual consistency) and migration leadership (strangler patterns, domain decomposition)
Event contract governance (Avro/Protobuf with schema registry), DLQ processing strategies, and data replay tooling
OSS contributions, technical talks, patents, or published design writing

Responsibilities

Lead the technical roadmap to modernize the ZIA control plane using API-first, domain-aligned microservices with clear service ownership and multi-region architectures (active-active, failover, disaster recovery)
Design, develop, and operate low-latency, event-driven services end to end (API through storage) utilizing resilience patterns, progressive delivery (blue/green, canary, feature flags), GitOps workflows (Argo CD/Flux), and safe rollout/rollback strategies
Drive engineering excellence through standards, code reviews, testing, paved-road patterns, API and event contract governance (versioning, consumer-driven contracts, schema registry, DLQs), and cross-functional partnerships across Product, SRE, Security, Data, and Architecture while representing designs in customer/partner forums
Own service reliability and operability end to end, including SLI/SLOs (p95/p99 latency, availability), SLO-based alerting, error budgets, incident response, postmortems, and advanced observability with distributed tracing (OpenTelemetry, Jaeger/Tempo) and structured logging
Embed microservices security (OAuth2/OIDC, mTLS, service-to-service authZ, KMS/Vault), supply-chain integrity (SBOM, SLSA), performance optimization, capacity planning, and cloud spend/autoscaling across AWS primitives