Senior AI Product Manager, Evals & Annotations
About The Position
Arize is seeking a highly technical, product-minded Senior AI Product Manager to own its Evals & Annotations product area. The role owns the end-to-end evaluation lifecycle, including eval creation and execution, human review, annotation workflows, analysis, and iteration. The Product Manager will shape a cohesive developer experience for evaluating AI applications and increasingly autonomous or managed agents, including opportunities in the Dynatrace ecosystem. The role combines product strategy and roadmap ownership with hands-on exploration of workflows, prototyping, and close collaboration with Design and Engineering. It involves technically complex work across evaluation execution, backend systems, orchestration, and data infrastructure, as well as frequent customer conversations to understand needs and validate direction. The Product Manager will work directly with Arize's cofounder/CPO and cross-functional teams, making prioritization decisions and driving products from problem definition through launch, adoption, and iteration. Arize describes itself as a remote-first company, with offices in New York City and the San Francisco Bay Area for employees who wish to work in person. Employees elsewhere receive a monthly work-from-home stipend for coworking spaces.
Requirements
- Significant product management experience independently owning technically complex, developer-facing products; exact years of experience are flexible for candidates with exceptional technical depth, product judgment, and ownership.
- Strong understanding of LLM evaluations and AI development workflows, including automated evaluations, human feedback, experimentation, and developer iteration on AI applications and agents.
- Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products, and a solid understanding of backend systems and software architecture.
- Strong technical depth and hands-on engineering ability, including debugging issues, navigating code, making small fixes or pull requests, building prototypes or proofs of concept, proposing technical implementations, and reasoning through tradeoffs.
- Excellent product and UX instincts, including the ability to define and prototype core product experiences and work effectively with designers and design engineers.
- Strong ownership, initiative, and bias toward action; comfort navigating ambiguity, making tradeoffs, and working hands-on without a fully defined roadmap.
- Collaborative, low-ego approach and ability to engage in constructive debate with founders, engineers, and designers in an environment where responsibilities overlap.
- Experience working closely with technical customers, with strong communication skills and the ability to move between strategy and execution.
Nice To Haves
- Direct experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration.
- Strong backend or platform engineering background, particularly in distributed systems, workflow orchestration, or data infrastructure.
- Hands-on programming experience and comfort working across backend and frontend technologies.
- Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback-driven improvement workflows.
- Previous engineering experience or substantial hands-on software development experience.
- Early-stage or high-growth startup experience, particularly in founder-led environments.
Responsibilities
- Own the Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle.
- Set product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development.
- Develop and prototype evaluation workflows and partner with Design to deliver intuitive, polished product experiences.
- Shape evaluation products for autonomous and managed agents, including opportunities in the broader Dynatrace ecosystem.
- Work with Engineering on evaluation execution, backend systems, orchestration, and data infrastructure; contribute to implementation discussions and technical tradeoffs.
- Speak with customers regularly, typically multiple times each week, to understand evaluation practices, uncover problems, and validate product direction.
- Collaborate with cofounders and cross-functional teams on product direction, design exploration, and prioritization.
- Make prioritization decisions across competing demands and drive products from problem definition through launch, adoption, and iteration.
Benefits
- Competitive equity package
- Medical, dental, and vision coverage
- 401(k) plan
- Unlimited paid time off
- Generous parental leave plan
- Mental and wellness support
- Monthly work-from-home stipend for coworking spaces for employees outside New York City and the San Francisco Bay Area
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
