Principal AI Software Engineer
About The Position
Palo Alto Networks’ Information Technology team secures the company’s digital landscape and serves as the first customer of its cybersecurity products. The Principal AI Software Engineer will be a hands-on technical leader advancing the company’s Generative AI transformation by architecting enterprise-grade internal and external business applications and intelligent, agentic workflows for Go-To-Market (GTM) and customer experience processes. The role leads the design of a proxy-first AI ecosystem, including a unified API and AI Gateway that abstracts LLM providers and supports centralized access and security controls. It also establishes GenAI observability, cost and performance optimization, feedback loops, multi-agent systems, business workflow automation, and evaluation frameworks. The engineer will implement safety guardrails and defenses against PII exposure, offensive content, prompt injection, and other attempts to manipulate model instructions. The position calls for extensive experience in software architecture and development, expert Python skills, hands-on LLM application development, and technical leadership. Palo Alto Networks describes its work model as primarily office-based, with flexibility when needed.
Requirements
- Bachelor’s degree or equivalent experience in Computer Science or a related field.
- 10+ years of IT system architecture, design, and development experience; alternatively, 8+ years with a master’s degree or 5+ years with a Ph.D.
- Proven technical leadership in Agile/Scrum environments, delivering large-scale implementations on aggressive timelines.
- Expert-level proficiency in Python.
- Extensive hands-on experience building applications using LLMs such as Llama 3, GPT-4, and Anthropic models.
- Deep experience with multi-agent systems, Agent-to-Agent (A2A) communication, and orchestration frameworks.
Nice To Haves
- Experience partnering directly with business stakeholders to design workflows for GTM, sales, or support operations.
- Familiarity with Java or Node.js for enterprise integration.
- Proficiency with AI tracing and evaluation tools such as LangSmith, Arize, HoneyHive, or custom OpenTelemetry implementations.
- Experience establishing Golden Datasets for ground-truth testing and conducting comparative analysis across models.
Responsibilities
- Architect and implement a proxy-first AI ecosystem and AI Gateway that allows LLM providers to be switched without refactoring core application code.
- Build a unified API endpoint that abstracts LLM providers and centralizes access and security controls.
- Implement GenAI observability pipelines to capture trace-level data, prompt inputs and outputs, and model latency.
- Optimize performance and costs by routing simpler queries to cost-effective models.
- Use observability feedback to identify bottlenecks, latency issues, and model drift in real time.
- Lead development of multi-agent systems and Agent-to-Agent (A2A) workflows.
- Apply GenAI to GTM business logic, automating workflows such as customer support processes and entitlement platforms.
- Develop LLM-as-a-Judge evaluation methods to assess the correctness, tone, and helpfulness of system outputs.
- Build guardrails to prevent PII exposure, offensive content, and prompt injection attacks.
- Develop adversarial defenses, including keyword filtering, behavioral analysis, and adversarial training, to protect model instructions.
Benefits
- Compensation may include restricted stock units and a bonus.
- Employee benefits are referenced in the job posting, but specific benefits are not listed.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
