Senior Staff Machine Learning Engineer

Zscaler•Santa Clara, CA

1d•Remote

About The Position

We are looking for a Senior Staff Machine Learning Engineer to join our team. This is a remote (USA) role, reporting to the Manager AI Platform and Data Science in the AI Platform and Data Science department. This team’s mission is high-fidelity risk identification in customer data, with goals to catch all threats while minimizing noise. To achieve this, you will develop solutions applying data analysis and threat-research while leveraging AI, machine learning, and data engineering to build quality, data-driven components to automate security analysis.

Requirements

Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
8+ years of professional Python development with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, incident debugging in live distributed systems, and familiarity with cloud and infrastructure-as-code in an AWS environment
Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects

Nice To Haves

Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with risk analysis teams
Data engineering experience building and operating pipelines over large-volume event data using SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto), schema evolution, backfills, and data quality validation

Responsibilities

Translate risk identification methods into agent logic, understanding the benefits and limitations of agents and ensuring quality across risk analysis, explanations, and recommendations
Collaborate closely with threat-research to understand data, threats, and their approach to developing security heuristics
Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
Follow a data-driven quality approach to threat detection, including backtesting, balancing precision vs recall, tuning, and quality control
Deploy and monitor your solutions in production within our CI/CD framework