hour

24-Mag•New York, NY

2d•Remote

About The Position

We are sharing a specialised part-time consulting opportunity for experienced Product Managers and Product Owners to contribute domain expertise to an advanced AI development project focused on evaluating product-related content, specifications, and written outputs. This is an evaluation and analysis engagement rather than a traditional product delivery role. Selected professionals will assess AI-generated responses across product-management scenarios, evaluate whether outputs satisfy specific requirements and audience needs, and provide detailed written reasoning that helps improve the quality of advanced AI systems. No prior AI experience is required.

Requirements

Professional experience as a Product Manager, Product Owner, or certified Product Owner with a strong background in cross-functional product development
Deep practical understanding of product management and product ownership rather than project, programme, delivery, or scrum-management responsibilities
Extensive experience authoring or evaluating specifications, PRDs, acceptance criteria, release notes, decision documents, or comparable product materials
Strong ability to determine whether written content is appropriate for executives, technical stakeholders, and end users
Excellent written communication skills and native-level English fluency
Demonstrated ability to provide clear, detailed, and actionable feedback on technical or user-facing documentation
Strong attention to detail, critical-analysis skills, and professional judgement
Ability to work independently and reliably within remote, asynchronous project environments
Candidates whose primary background is project management, programme management, delivery management, or scrum leadership are not the intended profile for this engagement

Nice To Haves

Familiarity with AI model evaluation, RLHF, annotation, or structured content-review workflows is advantageous but not required

Responsibilities

Assess and compare AI-generated responses to product-management and product-ownership prompts
Review outputs including specifications, release notes, user-facing copy, stakeholder communications, and related product documentation
Score responses based on clarity, tone, usefulness, instruction adherence, and suitability for the intended audience
Determine whether generated content adequately addresses the underlying product requirement or task
Identify responses that appear fluent or complete but fail to solve the actual problem presented
Evaluate product specifications, PRDs, acceptance criteria, release notes, decision documents, and similar materials
Assess whether documentation is sufficiently clear, complete, structured, and actionable
Identify factual inaccuracies, internal inconsistencies, missing requirements, and unsupported assumptions
Determine whether responses remain within the constraints established by the original prompt
Apply professional product judgement when assessing the quality and practicality of written outputs
Write detailed rationales explaining the reasoning behind evaluation scores
Clearly document why particular responses succeed or fail against stated requirements
Distinguish between surface-level writing quality and genuine usefulness to executives, technical stakeholders, or end users
Provide precise, actionable written feedback that can support improvements in model performance
Apply consistent evaluation standards across large volumes of content
Work independently within structured remote evaluation workflows
Collaborate asynchronously with project managers, reviewers, and other expert contributors
Follow project-specific guidelines, rubrics, and quality standards
Resolve ambiguous evaluation cases through clear written reasoning and professional judgement
Help maintain consistency and accuracy across product-focused AI evaluation datasets