Senior Psychometrician

NBME•Philadelphia, PA

35d•$138,688 - $173,360•Remote

About The Position

NBME's Psychometrics and Data Analysis (PADA) unit is seeking an experienced Senior Psychometrician to join its team. This position will apply advanced measurement expertise to support NBME's operational assessment programs, research initiatives, and continued advancement of psychometric practices. The role involves leading and contributing to applied research, planning and conducting psychometric analyses for operational examination programs, applying expertise to develop and evaluate responsible uses of artificial intelligence and emerging analytical methodologies, advancing the educational measurement field through presentations and publications, assessing current psychometric processes for improvements including AI and automation, and developing data visualizations. The Senior Psychometrician will collaborate with internal professionals and external clients and subject matter experts. This role is primarily remote with occasional onsite attendance required. NBME is committed to innovation and improving how it fulfills the evolving needs of the health care community by recruiting and empowering talented individuals from diverse backgrounds.

Requirements

Strong foundation in psychometric theory and its application to operational assessment, including expertise in item response theory (IRT), classical test theory (CTT), statistical methods, test development practices, and research design.
Advanced knowledge of measurement methodologies such as equating, scaling, generalizability theory, Bayesian methods, and/or standard setting.
Demonstrated ability to translate complex assessment results into clear and meaningful graphical displays that support appropriate interpretation, decision-making, and action.
Knowledge of or experience with natural language processing, process-data analytics, formative assessment methodologies, and related emerging approaches.
Strong technical and computational skills, including experience using statistical and psychometric software and programming tools such as calibration software, SAS, Python, and/or R.
Excellent written and verbal communication skills, with the ability to explain technical measurement concepts to audiences with varying levels of psychometric expertise.
Strong analytical and problem-solving skills, including the ability to identify data, methodological, and operational issues and develop appropriate solutions.
Excellent interpersonal and collaboration skills, with demonstrated success working with colleagues and stakeholders from diverse professional, technical, and educational backgrounds.
A minimum of 5 years of professional experience in a Psychometrician, Measurement Scientist, or comparable applied measurement role is required.
Experience supporting operational testing or assessment programs is strongly preferred.
A doctoral degree in Educational Measurement, Psychological Measurement, Statistics, or a closely related quantitative field is required.

Nice To Haves

Experience or a demonstrated interest in using artificial intelligence and emerging analytical approaches to support operational assessment processes.

Responsibilities

Lead and contribute to applied research focused on enhancing examination design, psychometric methodologies, and analytical procedures.
Plan, conduct, and evaluate psychometric analyses supporting operational examination programs, including scoring, equating, scaling, and score reporting activities.
Apply psychometric and measurement expertise to develop and evaluate responsible uses of artificial intelligence, machine learning, and emerging analytical methodologies in operational assessment processes.
Advance the educational measurement field through professional presentations, publications, and research aligned with both individual and organizational research priorities.
Assess current psychometric processes and methodologies to identify opportunities, including appropriate uses of AI and automation, to improve the accuracy, efficiency, and effectiveness of examination processing.
Develop and evaluate data visualizations and graphical displays that clearly communicate assessment information through stakeholder reports, examinee feedback, and interactive data tools.