Rubric Engineering and Measurement Specialist

Child Mind Institute•New York, NY

8h•Hybrid

About The Position

As the Rubric Engineering and Measurement Specialist at the Child Mind Institute, you will lead the implementation and validation of evaluation frameworks for assessing both AI systems in clinical and mental health research contexts, and the symptoms of their users. You will own the distillation of clinical definitions into evaluation rubrics, combining applied clinical science with evaluation operations to ensure that rubrics are clinically grounded and operationally scalable. You will work closely with both the clinical and data science teams, ensuring knowledge is effectively operationalized to the satisfaction of both specializations. Supervising a small team of data science and software professionals, you will establish reliability standards, oversee the design of metrics for use in multi-rater evaluation systems, and ensure scientific rigor through quality assurance, versioning, and comprehensive documentation.

Requirements

PhD, PsyD, MD, or equivalent experience in measurement science, psychometrics, biostatistics, clinical psychology, epidemiology, or related field
8+ years of applied evaluation experience in clinical research, health outcomes research, or psychometric assessment
5+ years of experience with patient evaluation, clinical rating scales, or performance assessment frameworks
5+ years of experience in at least one scientific programming language (e.g., Python/R, Matlab) and related toolboxes or frameworks (e.g., Tidyverse, Scipy, Sklearn, Polars, Pytorch) is required. Experience can be concurrent with education.
5+ years of experience with applied statistics, including reliability analysis, classical test theory, item response theory, or factor analysis.
Demonstrated leadership of multi-rater or benchmark systems.

Responsibilities

Translate clinical symptom evaluations and manualized therapeutic protocols into reliable, consistent, precise, and scorable evaluation criteria.
Design and implement metadata standards and protocols for multi-rater clinical evaluation systems, enabling crowd-sourced ratings from diverse clinical experts.
Set reliability standards and adjudication logic for resolving disagreements among clinical annotators, ensuring consistency across large-scale annotation efforts.
Establish and enforce quality assurance processes, versioning systems, and comprehensive documentation for all evaluation rubrics and measurement frameworks.
Contribute to scientific publications, white papers, and technical documentation of measurement methodologies and validation studies.
Collaborate with clinical leadership to ensure evaluation criteria accurately reflects clinical best practices, safety standards, and clinician feedback.
Supervise and mentor a small team of data analysts and engineers, providing guidance on evaluation methodologies, statistical analysis, and rubric implementation.
Collaborate with other technical project stakeholders, to ensure that technical definitions and requirements of all curated rubrics and resources are being met.