Engineering Product Manager

Cisco•Milpitas, CA

2d•$194,600 - $285,700

About The Position

The Cisco AI Software & Platform Group incubates and delivers mission-critical Generative AI and platform infrastructure across Cisco's global product portfolio. Our group builds foundational evaluation, benchmarking, and observability platforms that empower engineering teams to deploy autonomous AI agents and LLM-powered systems safely, reliably, and with verified accuracy at enterprise scale. As an Engineering Product Manager you will lead the strategy, architecture, and organizational adoption of Cisco's end-to-end evaluation platform:

Requirements

Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field with 12+ years of product management experience, OR Master's degree with 8+ years, OR PhD with 5+ years of experience.
Prior experience leading product management for AI/ML platforms, model evaluation frameworks, Large Language Model (LLM) infrastructure, or data-intensive software products.
Applied background designing and operationalizing offline benchmarking systems, automated evaluation judges, or real-time production quality monitoring pipelines.
Previous experience defining risk-based launch criteria, release-gating frameworks, and regression mitigation workflows for software or AI systems.
Proven experience driving multi-team platform adoption across matrixed engineering and product organizations.

Nice To Haves

Direct experience evaluating Generative AI, Retrieval-Augmented Generation (RAG) pipelines, and autonomous multi-agent systems.
Familiarity with modern AI evaluation toolkits (e.g., DeepEval, Ragas, LangSmith, TruLens) and enterprise telemetry architectures.
Prior experience operating across enterprise SaaS, cloud infrastructure, or hyperscaler environments.
Demonstrated background in statistical evaluation methodologies, automated LLM-as-a-judge calibration, and human-in-the-loop annotation workflows.
Practical experience with online evals for agent systems: observing production traces and trajectories to diagnose failure patterns and decide which techniques to apply to make the agent work better.

Responsibilities

Define AI Evaluation Platform Strategy: Architect the product vision, staged roadmap, and platform architecture for Cisco's unified offline benchmarking suites, automated LLM judges, and continuous production evaluation systems.
Operationalize Quality & Safety Standards: Convert ambiguous quality standards (e.g., factual accuracy, hallucination mitigation, safety policies, agentic tool-use fidelity) into objective, repeatable test suites and structured human-in-the-loop review pipelines.
Establish Release Gating & Quality Bars: Establish pre-deployment quality bars, risk-based release criteria, and automated regression-detection loops to govern AI model and application rollouts.
Bridge Offline & Production Telemetry: Connect offline model evaluations directly with live telemetry, product analytics (Amplitude), and real-time user feedback loops to rapidly detect and remediate performance regressions.
Drive Multi-Team Platform Adoption: Partner deeply with research scientists, engineering leaders, and product managers across business units to drive organization-wide adoption of standardized evaluation frameworks.

Benefits

medical, dental and vision insurance
a 401(k) plan with a Cisco matching contribution
paid parental leave
short and long-term disability coverage
basic life insurance
10 paid holidays per full calendar year, plus 1 floating holiday for non-exempt employees
1 paid day off for employee’s birthday, paid year-end holiday shutdown, and 4 paid days off for personal wellness determined by Cisco
16 days of paid vacation time per full calendar year, accrued at rate of 4.92 hours per pay period for full-time employees (non-exempt)
flexible vacation time off program (exempt)
80 hours of sick time off provided on hire date and each January 1st thereafter, and up to 80 hours of unused sick time carried forward from one calendar year to the next
Optional 10 paid days per full calendar year to volunteer
annual bonuses (for non-sales roles)
performance-based incentive pay (for sales roles)