Principal Software Automation/Test Engineer
About The Position
Palo Alto Networks is hiring a Principal Software Automation/Test Engineer to lead test architecture and automation strategy for Cortex Cloud services. The role focuses on building scalable, maintainable automation frameworks and tooling adopted across engineering teams, and creating complex integration environments involving multi-cloud accounts, Kubernetes clusters, cloud connectors, data pipelines, scanners, brokers, and third-party integrations. The engineer will embed quality gates, test orchestration, ephemeral environments, and release validation into CI/CD; lead system, scale, and chaos testing for distributed services; and troubleshoot complex failures across services, infrastructure, and cloud providers. The role also involves developing AI agents and LLM-powered testing tools, defining validation approaches for AI-driven product features, establishing quality metrics, raising engineering standards, mentoring senior and staff engineers, and influencing technical direction across development, QA, and DevOps teams. The position is primarily office-based, with flexibility when needed. Compensation depends on qualifications, experience, and work location. The posted annual base salary range is $156,400 to $253,000; compensation may also include restricted stock units and a bonus. Immigration sponsorship is not available for this role.
Requirements
- 10+ years of software engineering experience, including at least 6 years focused on test automation, quality engineering, or developer productivity for large-scale distributed systems or SaaS.
- Strong programming skills in at least one of Python, Go, or Java, with the ability to write production-quality code.
- Proven experience architecting and building test automation frameworks from scratch and scaling them across multiple teams.
- Deep hands-on experience with CI/CD systems and practices, including pipeline design, test parallelization, and environment provisioning.
- Strong cloud knowledge across GCP, AWS, and Azure, including compute, networking, IAM, storage, managed data services, and cloud-native security concepts.
- Experience with containers and orchestration, including Docker and Kubernetes/GKE/EKS/AKS, and Infrastructure as Code, including Terraform and Helm.
- Extensive experience with system, integration, end-to-end, performance, and scale testing of microservices and data pipelines, including REST/gRPC APIs, message queues such as Pub/Sub or Kafka, SQL and NoSQL stores, and BigQuery or similar.
- End-to-end domain expertise across enterprise cybersecurity systems to lead quality and system validation strategy.
- Excellent troubleshooting skills across application, infrastructure, network, and cloud-provider layers, using observability tools such as Cloud Logging, Grafana, and tracing.
- Ability to learn new, complex domains quickly and deliver in ambiguous, fast-moving environments.
- Strong communication skills and ability to influence technical decisions across organizations without direct authority.
- BS/MS in Computer Science or Engineering, or equivalent practical experience.
Nice To Haves
- Hands-on experience building AI agents or LLM-powered applications, such as with LangChain, LangGraph, MCP, OpenAI/Anthropic/Gemini APIs, RAG, tool calling, or multi-agent orchestration.
- Experience applying AI/ML to engineering workflows such as automated test generation, intelligent failure triage, or log and anomaly analysis.
- Experience testing or evaluating AI/LLM-based product features, including evaluation frameworks, guardrails, and non-deterministic output validation.
- Experience with UI automation frameworks such as Playwright or Selenium.
- Experience with security, resilience, or chaos engineering practices.
Responsibilities
- Own test architecture and define and drive the automation strategy for Cortex Cloud services.
- Design, implement, and maintain scalable, maintainable test frameworks and tooling in Python, Go, Java, or TypeScript for adoption across engineering teams.
- Learn unfamiliar systems and build and automate complex integration environments involving multi-cloud accounts, Kubernetes clusters, cloud connectors, data pipelines, scanners, brokers, and third-party integrations.
- Integrate quality gates, test orchestration, ephemeral test environments, and release validation into CI/CD pipelines.
- Lead system, scale, and chaos testing for high-throughput distributed services; identify bottlenecks, race conditions, and failure modes.
- Troubleshoot complex test and production-like failures across services, infrastructure, and cloud providers, and lead root-cause analysis using logs, metrics, traces, and data-store queries.
- Design and build AI agents and LLM-powered tools for test generation, failure triage, flaky-test detection, log analysis, and environment self-healing.
- Define validation approaches for AI-driven product features.
- Raise engineering standards through design reviews, code reviews, and best practices.
- Mentor senior and staff engineers and influence technical direction across development, QA, and DevOps teams.
- Define quality metrics, including coverage, escape rate, pipeline health, flakiness, and MTTR, and use them to improve release quality and speed.
- Lead overall quality and system validation strategy for enterprise cybersecurity systems.
Benefits
- Restricted stock units may be included in compensation.
- A bonus may be included in compensation.
- Employee benefits are referenced, but specific benefits are not listed in the job description.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
