Principal Product Manager, Agent Intelligence

Equinix • Toronto, ON

5d • $158,000 - $238,000

About The Position

Equinix is seeking a Principal Product Manager to own Agent Intelligence: the evaluation, observability, and feedback systems that measure agent performance, identify failure causes, and determine whether changes improve results. The role will establish quality standards for agents across Equinix and build the platform to enforce them. It is intended for someone with experience building or operating evaluation or observability systems for production ML or LLM systems at scale, and who can investigate system traces and root causes directly.

The position owns the roadmap for offline and online evaluation, including golden datasets, scenario and regression suites, calibrated LLM-as-judge pipelines, human review, and evaluation gates integrated into CI/CD. It covers measurement of component and end-to-end quality, reliability, observability, drift, cost, and adoption across agent workflows. The role also converts production feedback and confirmed failures into tests, requirements, or training and tuning data, and leads cross-functional work to improve agent quality while protecting data boundaries and meeting regulatory and confidentiality requirements.

In the first year, the successful candidate will establish a shared evaluation platform and golden datasets for core agent workflows, put evaluation gates before every agent release, deliver end-to-end tracing across multi-agent workflows, and make production failures into regression tests within days. The position is full-time. The targeted annual base pay range is USD 158,000–238,000 for Dallas, Texas, or CAD 158,000–236,000 for Toronto, Canada. Bonus eligibility and possible equity are separate from the stated base pay.

Requirements

• At least 7 years across product management and engineering, including at least 3 years as a product manager for technical platform, infrastructure, data, or ML products.
• A systems engineering background, with several years building or operating distributed systems, data platforms, infrastructure, or ML systems as an engineer, or equivalent hands-on depth.
• Demonstrated experience building and operating evaluation, quality, or observability systems for production ML or LLM products at scale, with results to show.
• Direct experience with multi-component systems such as agents, tool use, retrieval pipelines, or orchestrated services, including identifying which component caused failures.
• Strong evaluation judgment, including knowing when to use code-based checks, LLM judges, or human review; calibrating judges against experts; and recognizing when metrics do not reflect quality.
• Hands-on data skills, including writing SQL, reading traces and logs, and prototyping evaluations or analyses in Python or a similar language.
• Ability to lead without authority across engineering, data science, and business teams and clearly explain trade-offs to senior leaders.

Nice To Haves

• Experience with LLM evaluation and observability tools, tracing standards such as OpenTelemetry, and agent protocols such as MCP and A2A.
• Experience with retrieval, ranking, recommendation, confidence calibration, or learning-from-feedback systems.
• Experience with experimentation platforms and causal measurement.
• Experience with enterprise data such as CRM, billing, product usage, or network and infrastructure data.
• Experience with regulated customers or data governance, such as financial services, privacy, or confidentiality controls.
• A degree in computer science, engineering, or another quantitative field, or equivalent experience.

Responsibilities

• Define agent and end-to-end system quality standards covering correctness, source grounding, completeness, tool-use accuracy, workflow adherence, latency, cost, and business outcomes.
• Translate product requirements and acceptance criteria into measurable evaluations with clear thresholds and owners.
• Set consistent scoring standards and rubrics across agents, workflows, regions, and industries for both expert and automated evaluation.
• Own the roadmap for offline and online evaluation, including golden datasets, scenario and regression suites, calibrated LLM-as-judge pipelines, and human-in-the-loop review.
• Integrate evaluation-based quality gates into CI/CD so model, prompt, tool, or data changes must meet agreed thresholds before release.
• Measure reliability through repeated trials, consistency across runs, confidence intervals, and confidence calibration.
• Evaluate individual components and complete systems, including retrieval, entity resolution, ranking, tool and MCP calls, agent handoffs, and final outputs.
• Build trajectory- and trace-level evaluation to identify the component responsible for end-to-end failures and assess error propagation across handoffs.
• Define component contracts, including schemas, provenance, and confidence, so quality can be checked at each boundary.
• Partner with engineering to implement tracing and telemetry across agents, tools, and data sources so outputs can be traced to their inputs, sources, and decisions.
• Build monitoring and alerts for drift in inputs, data sources, models, and tool behavior.
• Provide leaders and product teams with trusted views of quality, reliability, cost, and adoption by agent, workflow, and segment.
• Analyze production traces, user edits, dismissals, ratings, and expert audits to find recurring failure modes, edge cases, and data gaps.
• Turn confirmed failures into regression tests, product requirements, or training and tuning data, and track them through resolution.
• Run experiments and A/B tests to measure whether changes improve accuracy, reliability, adoption, and business outcomes.
• Build evaluations and controls that enforce data boundaries and prevent inappropriate disclosure or cross-account exposure of confidential information.
• Partner with security, legal, and data governance teams to address regulatory and confidentiality requirements, including those for regulated industries.
• Set evaluation standards adopted by other product and engineering teams and influence teams outside the role's direct management.
• Write strategy documents, specifications, and executive briefs to align engineering, data science, sales, and leadership.
• In the first year, establish a shared evaluation platform and golden datasets, gate every agent release, deliver end-to-end tracing, and ensure production failures become regression tests within days.

Benefits

• Employee Assistance Program available to all employees.
• Health or healthcare insurance coverage.
• Life insurance.
• Disability insurance.
• Voluntary or optional benefit plans.
• Retirement or savings plans, including employer contributions where applicable.
• Accrued paid time off, vacation or personal time, and paid holidays.
• Employees are eligible for a bonus; equity may be offered depending on the position.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed