Executive Director, Natural Language Measurement

The College Board • New York, NY

96d • $148,000 - $225,000 • Remote

About The Position

As our Executive Director, Natural Language Measurement, you are a strategic measurement leader at the forefront of responsible AI in education. Energized by our mission to enhance educational and career opportunities for millions of students, you will shape the future of automated scoring and use of AI-powered feedback in large-scale assessments. You will set the vision and multi-year roadmap for natural language measurement across SAT, AP, CLEP, Accuplacer and future products, ensuring all applications are grounded in validity, fairness, and reliability. You will expand the College Board's innovation agenda in AI-enabled approaches that power personalized learning and feedback, reinforcing our leadership in responsible and equitable assessment.

Requirements

• At least 10 years of experience in psychometrics, assessment, or measurement-related fields.
• Strong record of applying measurement science in large-scale educational contexts.
• Deep expertise in measurement science and familiarity with AI/NL approaches.
• Proven ability to move AI powered natural language systems from research into operational use.
• Demonstrated success setting strategy and guiding innovation through cross-functional collaboration.
• At least 5 years of experience leading and managing direct teams with diverse expertise.
• A strategic and inclusive leadership style.
• A proven ability to drive performance and growth.
• A reputation as a thought leader at the intersection of psychometrics and AI/NL.
• Exceptional communication and organizational influence skills.
• Deep commitment to responsible use of artificial intelligence and to advancing equity in educational measurement.
• Doctorate in psychometrics, educational measurement, statistics, psychology, or a related field.

Responsibilities

• Shape and implement the multi-year strategy and roadmap for automated scoring and intelligent learning approaches across current flagship programs, such as SAT and AP, and emerging future products.
• Advance College Board's innovation agenda by expanding from automated scoring to broader AI-enabled solutions that support personalized learning and actionable feedback.
• Establish model-readiness criteria and guide release decisions for senior leadership to ensure high-quality, scalable deployment of natural language-driven solutions.
• Partner closely with Technology to shape development of guided scoring products and other intelligent learning innovations, advancing next-generation assessment capabilities.
• Lead scientific design, implementation, and evaluation of automated scoring and AI/NL applications, ensuring alignment with program and organizational goals.
• Apply psychometric and measurement principles to ensure all AI/NL-driven applications uphold validity, fairness, reliability, and equity.
• Guide implementation of initial automated scoring programs beginning in 2026.
• Establish and oversee operational processes that ensure quality, fairness, and reliability, including monitoring, human-in-the-loop procedures, and refinement of evaluation criteria.
• Build organizational readiness for adoption of new technologies, including criteria for model evaluation, quality assurance and responsible release decisions.
• Build and lead a team of psychometricians, language model experts, and related specialists, combining internal talent with new hires to deliver AI/NL priorities.
• Set vision and priorities for the team, track and manage progress to goals, and provide coaching and support to ensure team members meet and exceed goals.
• Cultivate an inclusive and high-achieving culture that enables all team members to live out College Board's Operating Principles effectively.
• Serve as College Board's lead subject-matter expert on AI powered natural language models.
• Collaborate with other College Board divisions, including close partnership with various program leaders, to ensure alignment on natural language-based initiatives and future assessment development.

Benefits

• A meaningful career, a supportive team, and a comprehensive package designed to help you thrive.
• Fair and competitive compensation, grounded in your qualifications, experience, impact, and the market.
• Open, transparent conversations about compensation, benefits, and what it's like to work at College Board.

What This Job Offers

• Job Type: Full-time
• Career Level: Executive
• Industry: Educational Services
• Education Level: Ph.D. or professional degree