Senior Psychometrician

NBME•Philadelphia, PA

32d•Remote

About The Position

NBME's Psychometrics and Data Analysis (PADA) unit is seeking an experienced Senior Psychometrician to join its team. This position will apply advanced measurement expertise to support NBME's operational assessment programs, research initiatives, and the continued advancement of psychometric practices. The role involves leading and contributing to applied research, planning and conducting psychometric analyses for operational programs, applying expertise to develop and evaluate responsible uses of AI and emerging analytical methodologies, advancing the educational measurement field through presentations and publications, assessing current psychometric processes for improvements, and developing data visualizations. The Senior Psychometrician will collaborate with internal teams and external clients and subject matter experts. This role is primarily remote, with occasional onsite requirements in Philadelphia.

Requirements

Strong foundation in psychometric theory and its application to operational assessment, including expertise in item response theory (IRT), classical test theory (CTT), statistical methods, test development practices, and research design.
Advanced knowledge of measurement methodologies such as equating, scaling, generalizability theory, Bayesian methods, and/or standard setting.
Demonstrated ability to translate complex assessment results into clear and meaningful graphical displays that support appropriate interpretation, decision-making, and action.
Knowledge of or experience with natural language processing, process-data analytics, formative assessment methodologies, and related emerging approaches.
Strong technical and computational skills, including experience using statistical and psychometric software and programming tools such as calibration software, SAS, Python, and/or R.
Excellent written and verbal communication skills, with the ability to explain technical measurement concepts to audiences with varying levels of psychometric expertise.
Strong analytical and problem-solving skills, including the ability to identify data, methodological, and operational issues and develop appropriate solutions.
Excellent interpersonal and collaboration skills, with demonstrated success working with colleagues and stakeholders from diverse professional, technical, and educational backgrounds.
A minimum of 5 years of professional experience in a Psychometrician, Measurement Scientist, or comparable applied measurement role is required.
Experience supporting operational testing or assessment programs is strongly preferred.
A doctoral degree in Educational Measurement, Psychological Measurement, Statistics, or a closely related quantitative field is required.

Nice To Haves

Experience or a demonstrated interest in using artificial intelligence and emerging analytical approaches to support operational assessment processes.

Responsibilities

Lead and contribute to applied research focused on enhancing examination design, psychometric methodologies, and analytical procedures.
Plan, conduct, and evaluate psychometric analyses supporting operational examination programs, including scoring, equating, scaling, and score reporting activities.
Apply psychometric and measurement expertise to develop and evaluate responsible uses of artificial intelligence, machine learning, and emerging analytical methodologies in operational assessment processes.
Advance the educational measurement field through professional presentations, publications, and research aligned with both individual and organizational research priorities.
Assess current psychometric processes and methodologies to identify opportunities, including appropriate uses of AI and automation, to improve the accuracy, efficiency, and effectiveness of examination processing.
Develop and evaluate data visualizations and graphical displays that clearly communicate assessment information through stakeholder reports, examinee feedback, and interactive data tools.
Work collaboratively across NBME with professionals in areas such as data analysis, test development, information technology, and program management.
Engage in external communication and collaboration with clients and subject matter experts representing a broad range of medical and health care disciplines.