Program Manager, Human Evaluations (Recommendations)

Netflix•Remote, OR

4d•$230,000 - $360,000

About The Position

Netflix is seeking an experienced Human Evaluations Program Manager to help measure and improve recommendation quality across the homepage and discovery experience. The role partners with Product Management and machine learning teams to build human evaluation programs that assess whether recommendations work for members and identify where they do not. The Program Manager will design and operate frameworks, tools, and workflows for consistent, clear, and thoughtful human evaluation of recommendation relevance, repetition, diversity, and whether recommendations make sense to members. The role owns execution from scoping and planning through vendor management, rater onboarding, calibration, and delivery. It also acts as a thought partner to Product Management, translating recommendation strategy into evaluation designs and establishing a shared understanding of recommendation quality across teams. Working across product, research, data science, UX, engineering, and vendors, this role develops scalable rubrics and workflows, tracks milestones and timelines, manages vendor capacity, and ensures consistent execution, timely delivery, and strong rater alignment. The ideal candidate combines rigorous program structure with flexibility, thrives amid ambiguity, and brings working fluency in film and TV content.

Requirements

At least 4 years of experience in human evaluations, data collection, labeling, or annotation operations in ML environments.
Track record of implementing process improvements or quality-control systems for data collection needs.
Experience managing human annotation vendors, raters, or data-labeling teams, including vendor performance and cost management.
Experience improving annotation-program operational efficiency, including throughput, turnaround, and cost.
Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols.
Working fluency in film and TV content, including genres, formats, and talent.
Proven ability to manage complex, cross-functional programs end to end, with strong program management and accountability for delivery.
Experience with human labeling platforms.
Excellent written and verbal communication skills.
Ability to synthesize feedback into clear recommendations and process improvements.
Familiarity with how rater subjectivity and bias affect taste-based judgments and how to account for them in evaluation design.
Strong organizational skills and executional focus, with the ability to track details while seeing the bigger picture.

Nice To Haves

Experience with personalization, recommender systems, search, or ranking evaluation.
Familiarity with how offline human evaluation complements online experimentation (A/B tests) and engagement metrics.
Experience evaluating at the list or page level, rather than only the individual-item level.
Experience partnering with Product Managers and Data Scientists on metric definition.

Responsibilities

Lead end-to-end execution of human evaluation and data operations initiatives, from intake and scoping through delivery.
Partner with Product Management to translate recommendation strategy and roadmap priorities into evaluation designs, success metrics, and timelines.
Manage human annotation vendors, including capacity planning, service-level agreements, quality performance, and escalations.
Improve operational efficiency across evaluation programs, including cost per task, throughput, turnaround time, and speed to inter-rater reliability.
Develop clear, scalable guidelines and scoring rubrics for consistent rater judgment, including subjective, taste-based questions.
Oversee rater onboarding, calibration, and quality assurance workflows.
Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality.
Anticipate and resolve bottlenecks and blockers.
Coordinate product, ML, data science, and vendor partners to align and execute evaluations at scale.