Engineering Product Manager

Cisco • Milpitas, CA

1d • $194,600 - $285,700

About The Position

The Engineering Product Manager will lead strategy, architecture, and organizational adoption of Cisco’s end-to-end AI evaluation platform. The platform supports unified offline benchmarking, automated LLM judges, and continuous production evaluation systems, helping engineering teams deploy AI agents and LLM-powered systems safely, reliably, and with verified accuracy at enterprise scale.

The role translates quality and safety standards—including factual accuracy, hallucination mitigation, safety policies, and agentic tool-use fidelity—into objective test suites and human-in-the-loop review pipelines. It establishes pre-deployment quality bars, risk-based release criteria, and automated regression detection. It also connects offline evaluations with production telemetry, product analytics, and user feedback, and drives adoption of standardized evaluation frameworks across business units in partnership with research scientists, engineering leaders, and product managers.

Requirements

• Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field and 12 or more years of product management experience; alternatively, a master's degree and 8 or more years of experience, or a PhD and 5 or more years of experience.
• Prior product management leadership experience with AI/ML platforms, model evaluation frameworks, LLM infrastructure, or data-intensive software products.
• Experience designing and operationalizing offline benchmarking systems, automated evaluation judges, or real-time production quality-monitoring pipelines.
• Experience defining risk-based launch criteria, release-gating frameworks, and regression-mitigation workflows for software or AI systems.
• Experience driving multi-team platform adoption across matrixed engineering and product organizations.

Nice To Haves

• Direct experience evaluating Generative AI, Retrieval-Augmented Generation (RAG) pipelines, and autonomous multi-agent systems.
• Familiarity with AI evaluation toolkits such as DeepEval, Ragas, LangSmith, and TruLens, and with enterprise telemetry architectures.
• Prior experience in enterprise SaaS, cloud infrastructure, or hyperscaler environments.
• Background in statistical evaluation methodologies, automated LLM-as-a-judge calibration, and human-in-the-loop annotation workflows.
• Practical experience with online evaluations for agent systems, including observing production traces and trajectories, diagnosing failure patterns, and selecting techniques to improve agent performance.

Responsibilities

• Define the AI evaluation platform vision, staged roadmap, and architecture, including offline benchmarking suites, automated LLM judges, and continuous production evaluation systems.
• Turn AI quality and safety standards into objective, repeatable test suites and structured human-in-the-loop review pipelines.
• Establish pre-deployment quality bars, risk-based release criteria, and automated regression-detection loops for AI model and application rollouts.
• Connect offline model evaluations with live telemetry, Amplitude product analytics, and real-time user feedback to detect and remediate performance regressions.
• Drive organization-wide adoption of standardized evaluation frameworks by partnering with research scientists, engineering leaders, and product managers across business units.

Benefits

• Medical, dental, and vision insurance, subject to plan eligibility rules.
• 401(k) plan with Cisco matching contribution.
• Paid parental leave.
• Short- and long-term disability coverage.
• Basic life insurance.
• Cisco restricted stock unit grants may be available, subject to eligibility and vesting conditions.
• Paid holidays and a floating holiday for non-exempt employees.
• Birthday paid day off, year-end holiday shutdown, and personal wellness days.
• Paid vacation or flexible vacation time off, depending on employee classification.
• Sick time off, with carryover of up to 80 unused hours.
• Additional paid time away may be requested for critical or emergency family issues.
• Optional paid volunteer days.
• Annual bonuses may be available for non-sales roles, subject to Cisco policies.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior