Engineering Product Manager
About The Position
The Cisco AI Software & Platform Group incubates and delivers mission-critical generative AI and platform infrastructure across Cisco’s global product portfolio. The group builds foundational evaluation, benchmarking, and observability platforms that help engineering teams deploy autonomous AI agents and LLM-powered systems safely, reliably, and with verified accuracy at enterprise scale. As Engineering Product Manager, you will lead the strategy, architecture, and organizational adoption of Cisco’s end-to-end evaluation platform. This includes defining the product vision, staged roadmap, and platform architecture for unified offline benchmarking suites, automated LLM judges, and continuous production evaluation systems. You will turn quality and safety expectations into objective test suites and human-in-the-loop review pipelines, establish release criteria and regression-detection processes, connect offline evaluations with production telemetry and user feedback, and drive adoption across teams and business units.
Requirements
- Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related technical field and 12+ years of product management experience; or a master’s degree and 8+ years; or a PhD and 5+ years of experience.
- Prior product management leadership experience for AI/ML platforms, model evaluation frameworks, LLM infrastructure, or data-intensive software products.
- Experience designing and operationalizing offline benchmarking systems, automated evaluation judges, or real-time production quality monitoring pipelines.
- Experience defining risk-based launch criteria, release-gating frameworks, and regression mitigation workflows for software or AI systems.
- Experience driving multi-team platform adoption across matrixed engineering and product organizations.
Nice To Haves
- Direct experience evaluating generative AI, Retrieval-Augmented Generation (RAG) pipelines, and autonomous multi-agent systems.
- Familiarity with AI evaluation toolkits such as DeepEval, Ragas, LangSmith, and TruLens, and with enterprise telemetry architectures.
- Prior experience in enterprise SaaS, cloud infrastructure, or hyperscaler environments.
- Background in statistical evaluation methodologies, automated LLM-as-a-judge calibration, and human-in-the-loop annotation workflows.
- Practical experience with online evaluations for agent systems, including observing production traces and trajectories to diagnose failure patterns and determine improvement techniques.
Responsibilities
- Define the AI evaluation platform vision, staged roadmap, and architecture for offline benchmarking, automated LLM judges, and continuous production evaluation.
- Translate quality and safety standards—including factual accuracy, hallucination mitigation, safety policies, and agentic tool-use fidelity—into repeatable test suites and structured human-in-the-loop review pipelines.
- Establish pre-deployment quality bars, risk-based release criteria, and automated regression-detection loops for AI model and application rollouts.
- Connect offline model evaluations with live telemetry, product analytics such as Amplitude, and real-time user feedback to detect and remediate performance regressions.
- Partner with research scientists, engineering leaders, and product managers across business units to drive organization-wide adoption of standardized evaluation frameworks.
Benefits
- Medical, dental, and vision insurance.
- 401(k) plan with Cisco matching contribution.
- Paid parental leave.
- Short- and long-term disability coverage.
- Basic life insurance.
- Cisco restricted stock unit grants may be available, subject to eligibility and continued-employment vesting conditions.
- 10 paid holidays per full calendar year and one floating holiday for non-exempt employees.
- Paid birthday day off, paid year-end holiday shutdown, and four paid personal wellness days.
- Non-exempt employees receive 16 days of paid vacation per full calendar year, accrued for full-time employees.
- Exempt employees participate in a flexible vacation time off program with no defined usage limit, subject to availability and business limitations.
- 80 hours of sick time provided on hire and each January 1, with up to 80 unused hours carried forward.
- Additional paid time away may be requested for critical or emergency family issues.
- Optional 10 paid volunteer days per full calendar year.
- Annual bonuses may be available for non-sales roles, subject to Cisco policies.
- Sales-plan incentive compensation may apply to sales roles.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
