Machine Learning Engineering Technical Leader

Cisco•Milpitas, CA

8d•$216,300 - $280,800

About The Position

Cisco’s Canvas AI team brings together AI researchers and software developers to build products and platforms. Canvas is an agentic workspace where customers’ IT teams and AI agents investigate and resolve issues across domains. The MLE Tech Lead sets the multi-quarter strategy for measuring agentic AI quality and owns evaluation from architecture through production operations. The role builds layered, trustworthy evaluation systems that combine automated checks, calibrated LLM judges, and expert human review; extends evaluation to live production traffic and simulation; and turns failures into validated fixes. The Tech Lead also drives shared measurement standards across partner teams, mentors engineers, and represents the domain to senior leadership.

Requirements

BA/BS with 8 or more years of relevant experience, MS with 6 or more years, or PhD with 3 or more years in Computer Science, Machine Learning, or a related quantitative field.
Experience taking machine-learning or LLM-based systems into production and owning them operationally, not only building prototypes.
Experience with Python and modern deep-learning frameworks such as PyTorch or TensorFlow.
Experience designing evaluation, measurement, or quality systems for ML or LLM products, including datasets, metrics, judges, benchmarks, or regression suites.
Experience setting technical direction across multiple teams through influence rather than reporting lines.

Nice To Haves

Practical experience designing and calibrating LLM-as-judge systems against human experts, including inter-annotator agreement and judge drift.
Experience evaluating agentic systems, including multi-step reasoning, tool selection, trajectory quality, and root-cause correctness.
Experience with production monitoring at scale, including asynchronous scoring of live traffic, drift detection, and actionable alerting.
Experience building reproducible test environments or synthetic data generation for domains where ground truth is scarce.
Experience integrating automated evaluation into CI/CD build pipelines and using it to gate releases.
Familiarity with telemetry, topology, and failure modes of enterprise networking and observability systems.
Experience with cost-aware inference methods such as routing, model selection, and optimization for production-scale evaluation.
Publications at top ML venues, open-source contributions, or public technical writing in the evaluation space.

Responsibilities

Own the evaluation roadmap and strategy for agentic AI quality, from architecture to production.
Design a layered evaluation architecture with deterministic checks on every pull request, model-based pre-release evaluation, and human-expert calibration.
Build and calibrate multi-model LLM-as-judge panels, tracking true-positive and true-negative rates separately.
Run the human-in-the-loop process with subject-matter experts, turning annotations into training signals for automated judges.
Score production traffic online, alert on regressions and drift, and convert real traces into offline test cases.
Lead simulation, synthetic data, and adversarial testing without consuming lab resources.
Close the loop from evaluation failure to clustered issue, candidate fix, and shadow or A/B validation.
Ensure trustworthy results through experiment snapshots, dataset versioning, coverage reporting, and cost tracking.
Drive common standards, data contracts, and judge criteria across partner teams through influence rather than direct authority.
Mentor engineers, set evaluation engineering standards, and represent the domain to senior leadership.

Benefits

Medical insurance
Dental insurance
Vision insurance
401(k) plan with Cisco matching contribution
Paid parental leave
Short- and long-term disability coverage
Basic life insurance
Paid holidays
Floating holiday for non-exempt employees
Paid birthday day off
Paid year-end holiday shutdown
Personal wellness days
Paid vacation time for non-exempt employees
Flexible vacation time off program for eligible exempt employees
Paid sick time
Additional paid time away may be requested for critical or emergency family issues
Optional paid volunteer days
Annual bonuses for eligible non-sales employees
Potential Cisco restricted stock unit grants