Senior Technical Product Manager – Agent Evaluation & Observability

American Express•New York, NY

2d•$123,000 - $215,250

About The Position

The Senior Technical Product Manager will lead the roadmap for shared tools and workflows that help teams test and improve agent-enabled customer experiences. The role focuses on measuring how effectively agents complete customer tasks, understanding failures, and using evidence to improve experiences. The position owns product roadmap, requirements, and adoption for shared evaluation tools and workflows, partnering with engineering and enterprise platform teams to build on existing capabilities. Domain product owners remain accountable for release decisions. The role includes developing agent evaluation and observability capabilities from user discovery through launch and ongoing improvement; defining representative journeys, prompt sets, expected tool behavior, outputs, scoring rubrics, and acceptance criteria with domain owners; and establishing automated evaluation, human review, and failure-diagnosis tools. It also defines requirements for traces, events, diagnostic views, and review tools that connect customer intent, tool or API calls, and observed outcomes where data access permits. The product manager collaborates with engineering, analytics, enterprise evaluation and observability teams, Identity, and domain owners to reuse instrumentation, standardize evidence, test policy-relevant behavior, and assess limitations caused by restricted partner-data access. The role defines measures such as task completion, correct tool use, response quality, failure and recovery rates, latency, and cost; clarifies what can be measured in controlled testing, partner integrations, and production; and uses findings to prioritize improvements. It also builds privacy and data governance into evaluation and telemetry requirements, including permitted data use, appropriate access, minimization, and retention, while clearly communicating uncertainty, release risks, and tradeoffs.

Requirements

6–8+ years of experience in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products.
Experience owning evaluation, testing, observability, analytics, or related technical products through launch, adoption, and iteration, with measurable user or business outcomes.
Practical understanding of AI or agent-enabled systems, including nondeterministic behavior, tool or API interactions, and the need for both automated tests and human judgment.
Ability to define metrics, interpret evidence, and identify sampling, coverage, or measurement limitations.
Familiarity with traces, logs, APIs, and technical troubleshooting.
Strong product discovery, roadmap, and prioritization skills, including translating ambiguous quality problems into usable tools and repeatable workflows.
Cross-functional leadership and clear communication, with the ability to balance customer outcomes, engineering constraints, and privacy or risk requirements.
Employment eligibility to work with American Express in the United States; the company will not pursue visa sponsorship.

Nice To Haves

Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring, or production failure analysis.
Familiarity with MCP, function calling, telemetry standards, or evaluation frameworks; ability to use code or queries to investigate evidence.
Experience building shared capabilities adopted by multiple teams or delivering products in financial services or another regulated industry.
Bachelor’s degree or equivalent experience.

Responsibilities

Own the roadmap for shared agent evaluation and observability capabilities, from user discovery and requirements through launch, adoption, and improvement.
Partner with domain owners to define representative journeys, prompt sets, expected tool behavior and outputs, scoring rubrics, and acceptance criteria.
Define requirements for automated evaluation, human review, and tools that help teams diagnose failures.
Define product requirements for traces, events, diagnostic views, and review tools that connect customer intent, tool or API calls, and observed outcomes where data access permits.
Work with engineering, analytics, and enterprise evaluation and observability teams to extend shared capabilities, reuse instrumentation, and standardize evidence.
Identify where limited access to partner data prevents a complete assessment.
Partner with Identity and domain owners to test policy-relevant behavior, including denied actions, revoked access, and required approvals.
Define meaningful measures, including task completion, correct tool use, response quality, failure and recovery rates, latency, and cost.
Define how testing and monitoring work across controlled test environments, end-to-end partner integrations, and production; clarify what each can measure and use findings to prioritize improvements with responsible teams.
Measure adoption and usefulness of evaluation workflows, including review effort, coverage, and time to identify problems; communicate limits of findings based on sampled data.
Translate measurement gaps into a prioritized backlog, clear requirements, and actionable acceptance criteria.
Recommend when to extend existing tools, integrate a solution, or build a missing capability.
Build privacy and data governance into evaluation and telemetry requirements, including permitted uses of partner-provided data, appropriate access, minimization, and retention.
Communicate uncertainty, release risks, and tradeoffs clearly.

Benefits

Competitive base salaries
Bonus incentives
6% Company Match on retirement savings plan
Free financial coaching and financial well-being support
Comprehensive medical, dental, vision, life insurance, and disability benefits
Flexible working model with hybrid, onsite, or virtual arrangements depending on role and business need
20+ weeks paid parental leave for all parents, regardless of gender, offered for pregnancy, adoption, or surrogacy
Free access to global on-site wellness centers staffed with nurses and doctors, depending on location
Free and confidential counseling support through the Healthy Minds program
Career development and training opportunities