Director, Free Response Rating Capability Analyst

College Board•Remote - Virginia, VA

26d•$80,000 - $130,000•Remote

About The Position

As the Director, FR Rating Capability Analyst, you will play a critical role in designing, building, and improving the operational and analytic capabilities that enable high-quality AP Reading scoring. This role sits at the intersection of scoring operations, data analysis, digital scoring systems, and process improvement. You will translate scoring performance insights into scalable tools, workflows, reporting frameworks, and operational improvements that strengthen scoring accuracy, efficiency, and reader effectiveness. You will collaborate closely with cross-functional partners to support continuous improvement across the AP Reading lifecycle and help scale digital scoring capabilities for the future.

Requirements

Expertise in operational analytics, scoring operations, education systems, or related fields with experience supporting large-scale operational environments.
Strong analytical and systems-thinking skills, with demonstrated ability to translate data into operational improvements and scalable solutions.
Experience working with large-scale datasets and reporting tools such as Excel, SQL, BI tools, or similar technologies.
Strong project management and prioritization skills with the ability to manage competing priorities across multiple stakeholders.
Excellent written and verbal communication skills, including the ability to simplify complex analyses into actionable insights.
Experience collaborating across cross-functional teams including operations, technology, product, analytics, and research partners.
Comfort operating in both strategic planning and hands-on execution environments.
Bachelor’s degree required.
Authorization to work in the United States for any employer.
Curiosity and enthusiasm for emerging technologies, including AI-driven solutions and digital tools.
Clear and concise communication skills, written and verbal.
A learner’s mindset and commitment to continuous growth and improvement.
A drive for impact and excellence through data-informed decision-making and problem-solving.
A collaborative and empathetic approach that fosters trust and shared success.

Nice To Haves

Experience in assessment, scoring operations, educational measurement, or education data systems strongly preferred.
Advanced degree preferred.

Responsibilities

Define and maintain the capability roadmap for FR rating operations, including tools, workflows, reporting structures, and performance measurement systems that support high-quality scoring at scale.
Translate operational needs and scoring insights into clearly defined business requirements, system enhancements, reporting frameworks, and process improvements.
Partner with Technology and Product teams to design, test, and implement enhancements to scoring platforms and data pipelines.
Develop scalable frameworks for evaluating scoring effectiveness, including consistency, accuracy, throughput, and reader-level performance metrics.
Establish and maintain expertise in AP exam item types, rubric structures, allowable response formats, and annual assessment changes that impact scoring operations.
Serve as a liaison across Scoring, Reader Management, Assessment Production, Psychometrics, and Technology teams regarding FR rating operational systems and processes.
Support the end-to-end AP Exam scoring workflow, including reader training, scoring operations, psychometric data collection, and operational reporting.
Partner with cross-functional teams to ensure scoring systems are configured appropriately for annual AP Reading operations across multiple subjects, forms, and item types.
Monitor scoring performance signals during live AP Reading and enable rapid operational interventions through data-driven insights.
Collaborate with Reader Management and Workforce Operations teams to align reader performance insights with recruitment, training, and retention strategies.
Support onboarding and training for stakeholders on scoring tools, scoring system processes, and calibration strategies.
Support AP Reader communications related to scoring platform usage, engagement portals, messaging, and operational processes.
Provide operational support during peak scoring periods and help resolve implementation or escalation issues related to scoring systems and workflows.
Develop analytic models, dashboards, and reporting tools that provide real-time and post-Reading insights into scoring performance.
Analyze scoring trends to identify operational risks, anomalies, reliability concerns, training gaps, and workflow inefficiencies.
Translate complex data into actionable insights and recommendations for scoring leadership and operational stakeholders.
Partner with Psychometrics and Research teams to align operational metrics with scoring validity and reliability standards.
Establish standardized reporting and postmortem analysis processes that support continuous improvement across Reading cycles.
Develop and maintain standardized processes, governance documentation, and operational frameworks for FR rating analytics and reporting.
Create and maintain cross-functional RACI documentation related to digital scoring platform deployment, configuration, and operational support.
Lead retrospective reviews following Reading cycles and synthesize findings into prioritized operational and capability improvements.
Drive automation and efficiency initiatives that reduce manual processes and improve operational accuracy and speed.
Identify and implement best practices across subjects to improve consistency in scoring operations and performance management.
Contribute to long-term strategy efforts supporting digital transformation and scaling of AP Reading capabilities.