Senior Machine Learning Engineer

Zscaler•Santa Clara, CA

3d•$115,500 - $165,000•Remote

About The Position

We are looking for a Senior Machine Learning Engineer to join our team. This is a remote (USA) role, reporting to the Manager AI Platform and Data Science in the AI Platform and Data Science department. This team’s mission is high-fidelity risk identification in customer data. Our goals are to catch all threats while minimizing noise. To do this, you will develop solutions applying data analysis and threat-research, while using AI, machine learning, and data engineering to build quality, data-driven components to automate security analysis.

Requirements

Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
3+ years of professional Python development, with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, and incident debugging in a live distributed system, with familiarity with cloud and infrastructure-as-code in an AWS environment
Senior-level ownership and leadership of projects, development, and testing

Nice To Haves

Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with these types of teams to understand risk analysis and investigation processes
Data engineering experience, including building and operating pipelines over large-volume event data: SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto, or similar), plus schema evolution, backfills, and data quality validation

Responsibilities

Translate risk identification methods into agent logic, understanding the benefits and limitations of agents, and how to ensure quality in different types of tasks, including risk analysis, explanations, and recommendations
Collaborate closely with threat-research to understand data and threats and their approach to developing security heuristics
Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
Follow a data-driven quality approach to threat-detection: including backtesting, balancing precision vs recall, tuning, and quality control
Deploy and monitor your solutions in production in our CI/CD framework