Principal Machine Learning Engineer (AI Agents)

Palo Alto Networks • Santa Clara, CA

83d • $175,000 - $215,000

About The Position

We are seeking a Principal Software Engineer with deep expertise in designing, building, and scaling AI-powered platforms. In this role, you will help shape the future of our agentic platform by leveraging advanced machine learning (ML) techniques to tackle complex, large-scale challenges and deliver impactful customer experiences. This role is located at our Santa Clara Headquarters Campus 3 days a week.

Requirements

• 8+ years in ML, data/analytics, and software engineering with production experience.
• Strong coding skills in Python and proficiency with SQL, including performance/scalability optimization.

Nice To Haves

• End-to-end experience designing and deploying RAG systems (indexing strategy, retrieval optimization, reranking).
• Expertise with LLMs and fine-tuning techniques (e.g., LoRA/QLoRA), prompt/agent design, and function-calling patterns.
• Familiarity with Google ADK (agents, long-term knowledge/memory) and LlamaIndex (graph construction, query engines).
• Strong background in NLP and/or recommender systems; experience with evaluation methods and dataset curation.
• Experience with microservices on GCP (e.g., GKE/Cloud Run, Pub/Sub, Vertex AI, CloudSQL/BigQuery) and real-time streaming.
• Ability to work independently and in cross-functional teams with excellent written and verbal communication.
• Hands-on experience with vector search and RAG frameworks.
• Proven track record deploying ML systems to production with CI/CD and observability.
• M.S. or Ph.D. in a technical field (or equivalent practical experience).

Responsibilities

• Lead the design, prototyping, and productionization of AI agent systems that solve complex user and business problems in cybersecurity applications.
• Own agent architectures end-to-end-from planning and orchestration to evaluation, deployment, and observability-while mentoring engineers and shaping our AI strategy.
• Design & build agentic systems: Architect workflows and POCs using frameworks such as Google ADK and LlamaIndex; implement tool use, function calling, and multi-step planning.
• Develop RAG pipelines (indexing, retrieval, reranking), code-interpreter/tool execution flows, and robust context management.
• Define evaluation suites for performance, efficiency, safety, and business alignment; analyze latency, quality, and cost trade-offs.
• Deploy models and agents to production; build scalable ML pipelines for batch and real-time/streaming use cases; implement monitoring and guardrails.
• Drive end-to-end delivery with modern CI/CD; automate testing, rollout, and experiment tracking.
• Partner with ML engineers, data scientists, and product to deliver roadmaps; mentor teammates and lead technical design reviews.
• Maintain clear specs and decision records; communicate complex concepts to technical and non-technical audiences.
• Contribute to AI product vision; incubate new AI initiatives and design microservices-based solutions on GCP.

Benefits

• Flexible work arrangements with in-person interactions encouraged.
• FLEXBenefits wellbeing spending account with over 1,000 eligible items.
• Mental and financial health resources.
• Personalized learning opportunities.
• Restricted stock units and a bonus.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Industry: Professional, Scientific, and Technical Services
• Education Level: Master's degree
• Number of Employees: 5,001-10,000 employees