Director, Psychometric Operations & Data Science

College Board

24d•$64,000 - $80,000•Hybrid

About The Position

The Director, Psychometric Operations & Data Governance, is expected to be key architect of CB-LATAM’s data ecosystem—ensuring that psychometric and operational data are managed ethically, governed responsibly, and leveraged to drive both operational excellence and strategic insight. In this role, you will define and implement the frameworks that safeguard the integrity, accessibility, and ethical use of CB-LATAM’s data assets. You will oversee the modernization of data architecture, strengthen data quality and compliance, and foster a culture of transparency and accountability around data use. You will design and maintain automated data pipelines that enable scoring, scaling, and reporting processes to run smoothly and reproducibly. You will be accountable for psychometric operations such as item calibrations, statistical analysis, and quality control to support scoring operations for current and future assessments. You will also play a central role in shaping how R&M integrates AI-enabled analytics and automation to streamline workflows and scale psychometric processes. By combining technical expertise with policy alignment and collaboration across R&D and Technology, you will help transform CB-LATAM into a data-driven, innovation-ready organization—ensuring that every data point supports fairness, validity, and impact.

Requirements

Master’s degree in Data Science, Computer Science, Statistics, Information Systems, Psychometrics, or related field.
8–12+ years of experience in data architecture, governance, analytics leadership, or high-stakes operational data environments.
Demonstrated experience owning production data systems with strict accuracy and compliance requirements.
Proven track record implementing data governance frameworks and automation strategies.
Strong technical fluency in SQL and at least one programming language (R or Python).
Experience designing or operating data workflows within AWS cloud infrastructure and working with structured and semi-structured data formats (e.g., JSON).
Working knowledge of IRT and CTT models, scaling, and equating procedures.
Strong executive communication and stakeholder alignment skills.
Ability to travel 1-2 times a year to College Board offices or on behalf of College Board business.
Bilingual proficiency in Spanish and English.
This position primarily involves sedentary work performed at a computer. The role requires the ability to sit or stand for extended periods, communicate effectively, and operate standard office equipment.
All roles at College Board require: A passion for expanding educational and career opportunities and mission-driven work
Curiosity and enthusiasm for emerging technologies, with a willingness to experiment with and adopt new AI-driven solutions and comfort with learning and applying new digital tools independently and proactively.
Clear and concise communication skills, written and verbal
A learner's mindset and a commitment to growth: welcoming diverse perspectives, giving and receiving timely, respectful feedback, and continuously improving through iterative learning and user input.
A drive for impact and excellence: solving complex problems, making data-informed decisions, prioritizing what matters most, and continuously improving through learning, user input, and external benchmarking.
A collaborative and empathetic approach: working across differences, fostering trust, and contributing to a culture of shared success
Resident in USA, Authorization to work in the United States for any employer

Nice To Haves

Experience deploying AI/ML models into operational workflows.

Responsibilities

Psychometric Operations Leadership (30%) Drive and safeguard the operational integrity of scoring, equating, calibration, and reporting data workflows.
Execute and monitor scoring cycles meet defined turnaround time, accuracy, and QA standards.
Design and implement reproducibility, documentation, and traceability standards across all psychometric processes.
Architect and implement modernization of legacy scoring and item analysis pipelines into relational or warehouse-based systems.
Conduct root-cause analysis for systemic scoring irregularities or data discrepancies.
Partner with Psychometricians to ensure form rotation, scaling, and calibration processes are technically sound and operationally resilient.
Enterprise Data Governance & Stewardship (25%) Serve as the designated Data Steward for psychometric and operational data within CB-LATAM.
Design and operationalize a unified governance framework covering metadata, retention, access control, and data lifecycle management.
Maintain comprehensive end-to-end data flow documentation across systems.
Ensure compliance with College Board governance policies, professional testing standards (AERA/APA/NCME), and applicable privacy regulations.
Establish governance KPIs and risk monitoring mechanisms.
Lead audit readiness efforts and documentation processes.
Data Architecture & Infrastructure Modernization (20%) Define and execute the roadmap for transitioning from manual, file-based workflows to automated, scalable infrastructure.
Partner with Technology to implement secure, cloud-aligned environments for psychometric and scoring data.
Establish standards for schema design, lineage tracking, validation routines, and version control.
Execute and validate migration of core datasets into governed architecture.
Design and validate technical design decisions for new pipelines and automation initiatives.
Automation, AI & Advanced Analytics (15%) Lead integration of AI-assisted validation and anomaly detection tools within scoring and integrity review workflows.
Establish model governance standards including documentation, benchmarking, and human-review protocols.
Identify automation opportunities that reduce manual effort while increasing consistency and quality.
Translate complex analytic outputs into executive-ready insights and operational recommendations.
Organizational Leadership & Influence (10%) Represent R&M in enterprise data architecture and modernization initiatives.
Mentor team members in governance, reproducible research practices, and advanced analytics.
Advise senior leadership on data risk, infrastructure investments, and operational priorities.
Promote a culture of accountability, data ethics, and continuous improvement.