Technical Product Manager, Pedagogy & AI Quality

Workshop Hiring • Boston, MA

8h • $130,000 - $150,000 • Hybrid

About The Position

Medley Learning is hiring a Technical Product Manager, Pedagogy & AI Quality to ensure its AI-powered instructional scaffolds meet high standards for quality and user experience. The role will develop quality rubrics for reading and writing support across multilingual learning, special education, and subject areas; expand quality metrics and evaluation systems; and establish review processes for AI outputs.

This hybrid product role combines pedagogy and AI work. The position involves writing and testing prompts, labeling and analyzing model outputs, reviewing traces and error reports, and working closely with engineering to deliver quality improvements. The role will apply research on scaffolding instructional materials for multilingual learners at different proficiency levels, as well as accommodations and supports for students with receptive and expressive language disabilities and neurodivergent students.

Working closely with the head of product and engineers, the Technical Product Manager will help ensure that teachers and students in partner districts receive high-quality scaffolds. Over time, the role will own the quality and usability of the growing scaffolds suite, prioritize product improvements and features, and support teachers and student outcomes.

Requirements

• Expertise in pedagogy, especially scaffolding literacy instruction for struggling readers, special education, and/or multilingual learner education, ideally gained through classroom teaching, intervention, instructional coaching, or curriculum design.
• Hands-on experience with AI evaluations, including human labeling, LLM-as-judge rubric design, defining AI metrics targets, and analyzing errors in LLM outputs.
• AI product fluency, including daily use of LLMs and experience designing complex AI systems for products.
• Experience defining product features and collaborating with engineers to deliver them, including communicating problems and solutions and working through technical tradeoffs and user-experience questions.
• Strong initiative, judgment, and writing and communication skills, including the ability to make data- and instinct-informed decisions and communicate with technical and non-technical audiences.

Nice To Haves

• Graduate degree and/or significant direct experience in special education, reading/literacy, ESL/ELD, or bilingual education.
• Familiarity with AI evaluation and observability tools such as LangSmith, Braintrust, Arize, or Langfuse, or with annotation platforms.
• Proficiency in a language other than English.
• Prior experience at an edtech company, or in content quality, assessment design, or psychometrics.
• Familiarity with accessibility standards such as WCAG and assistive technology.

Responsibilities

• Translate research-based literacy, special education, and multilingual learner instruction practices into clear quality standards for Medley scaffolds, and set targets for AI output quality.
• Build and maintain datasets, rubrics, and evaluation suites; monitor quality through regular error analysis and trace review using human labeling, LLM-as-judge, and rule-based evaluations.
• Write, test, and refine prompts; run model comparisons and regression tests; and make changes to ensure outputs meet target metrics and quality thresholds.
• Review educator feedback, support-ticket data, and product analytics, and conduct user research with students and teachers to inform product decisions.
• Maintain a prioritized backlog of AI quality and scaffold user-experience issues; write clear specifications; and work with engineering to scope, build, and verify fixes, product updates, and new features.
• Own the quality and usability of the scaffolds suite as it grows, prioritizing improvements and features that better support teachers and drive student outcomes.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed