Machine Learning Engineering Technical Leader

Cisco•Milpitas, MI

6d•$216,300 - $280,800

About The Position

Cisco’s Canvas AI team brings together AI researchers and software developers to build products and platforms. Canvas is an agentic workspace in which customers’ IT teams and AI agents investigate and resolve issues across domains. The Machine Learning Engineering Technical Leader will set the multi-quarter strategy for measuring agentic AI quality and own evaluation from architecture through production operation. The role builds layered, trustworthy evaluation systems that combine automated checks, calibrated LLM judges, and expert human review; extends evaluation to live production traffic and simulation; and ensures failures lead to validated fixes. The leader also establishes shared measurement standards across partner teams, mentors engineers, and represents the domain to senior leadership. Cisco lists a starting salary range of $216,300 to $280,800 for new hires in U.S. and/or Canada locations, excluding incentive compensation, equity, and benefits. Actual pay depends on factors including location, market conditions, skills, experience, qualifications, education, certifications, and training. Certain location-specific salary ranges are also provided. Eligible U.S. employees may receive Cisco restricted stock unit grants, subject to vesting conditions.

Requirements

Bachelor’s degree with at least 8 years of relevant experience, master’s degree with at least 6 years, or Ph.D. with at least 3 years in Computer Science, Machine Learning, or a related quantitative field.
Experience taking machine-learning or LLM-based systems into production and owning them operationally, beyond building prototypes.
Experience with Python and modern deep-learning frameworks such as PyTorch or TensorFlow.
Experience designing evaluation, measurement, or quality systems for ML or LLM products, including datasets, metrics, judges, benchmarks, or regression suites.
Experience setting technical direction across multiple teams through influence rather than reporting lines.

Nice To Haves

Practical experience building and calibrating LLM-as-judge systems against human experts, including inter-annotator agreement and judge drift.
Experience evaluating agentic systems, including multi-step reasoning, tool selection, trajectory quality, and root-cause correctness.
Experience with production monitoring at scale, including asynchronous scoring of live traffic, drift detection, and actionable alerting.
Experience building reproducible simulation or synthetic-data environments for domains where ground truth is scarce.
Experience integrating automated evaluation into CI/CD build pipelines and using it to gate releases.
Familiarity with networking, observability, or infrastructure domains, including enterprise network telemetry, topology, and failure modes.
Experience with cost-aware inference, including routing, model selection, and optimization to make evaluation viable at production volume.
Publications at top ML venues, open-source contributions, or public technical writing in the evaluation space.

Responsibilities

Own the roadmap and strategy for evaluating agentic AI quality, from architecture through production.
Design a layered evaluation architecture with deterministic checks on every pull request, model-based pre-release evaluation, and calibration by human experts.
Build and calibrate multi-model LLM-as-judge panels, tracking true-positive and true-negative rates separately.
Operate a human-in-the-loop feedback process with subject-matter experts, turning annotations into training signals for automated judges.
Score production traffic online, alert on regressions and drift, and convert real traces into offline test cases.
Lead simulation, synthetic data, and adversarial testing without consuming lab resources.
Connect evaluation failures to clustered issues and candidate fixes, then validate fixes through shadow testing or A/B testing.
Ensure trustworthy results through experiment snapshots, dataset versioning, coverage reporting, and cost tracking.
Drive common standards, data contracts, and judge criteria across partner teams through influence rather than direct authority.
Mentor engineers, establish evaluation engineering standards, and represent the domain to senior leadership.

Benefits

Medical insurance
Dental insurance
Vision insurance
401(k) plan with Cisco matching contribution
Paid parental leave
Short- and long-term disability coverage
Basic life insurance
Cisco restricted stock unit grants may be available, subject to eligibility and vesting conditions
10 paid holidays per full calendar year
One floating holiday for non-exempt employees
One paid day off for an employee’s birthday
Paid year-end holiday shutdown
Four paid personal wellness days
Non-exempt employees receive 16 vacation days per full calendar year, accrued at 4.92 hours per pay period for full-time employees
Exempt employees participate in a flexible vacation time-off program with no defined usage limit, subject to availability and business limitations
80 hours of sick time provided on hire and each January 1, with up to 80 hours of unused sick time carried forward
Additional paid time away may be requested for critical or emergency family-member issues
Optional 10 paid volunteer days per full calendar year
Annual bonuses may be available to non-sales employees, subject to Cisco policies
Sales employees may receive performance-based incentive pay under applicable Cisco plans