Assessment Designer & Learning Analyst

Mercor•San Francisco, CA

2d•$120,000 - $160,000•Onsite

About The Position

Mercor is seeking an Assessment Designer & Learning Analyst to build rigorous measurement systems and use data to understand what drives expert performance. This role focuses on designing assessments and certification frameworks to measure the skills of talent experts and internal teams, and analyzing the data to understand what predicts effectiveness and how programs should evolve. The position involves close collaboration with the Learning & Development team to analyze the relationship between instructional materials and assessments, and to make data-driven recommendations. The ideal candidate has a background in education, experience in high-accountability teaching environments, and has completed quantitative projects or theses, with a strong interest in the measurement and data aspects of education.

Requirements

Master's degree in Learning Sciences, Educational Psychology, Educational Measurement, Psychometrics, or a closely related field.
Coursework in quantitative research methods, psychometrics, and educational statistics.
Familiarity with classical test theory (CTT).
Genuine comfort with numbers and quantitative analysis.
Proficiency in item-level analysis: difficulty index, discrimination index, inter-rater reliability (Cohen's kappa, Krippendorff's alpha, ICC).
Ability to assess and report on assessment validity and reliability.
Experience analyzing relationships between variables: correlation, regression, and basic predictive modeling.
Fluent use of Excel or Google Sheets for data cleaning and summaries.
Basic proficiency in Python, STATA, or R for deeper analysis.
Ability to translate quantitative findings into plain-language recommendations for non-technical stakeholders.
1–2 years of experience in assessment design, educational research, learning analytics, or a related role.
Experience designing assessments with a clear theory of what is being measured.
Strong understanding of measurement principles: validity, reliability, and assessment quality.
Ability to connect data analysis to program meaning and impact.
Strong writing skills for clear communication to non-technical audiences.
Systems thinking ability to connect assessments to operational quality and expert performance.
Comfort with ambiguity and rapid iteration.

Nice To Haves

Experience with item response theory (IRT) or latent variable modeling.
Familiarity with data annotation, labeling, or AI evaluation workflows.
Experience in tech, AI/ML, or data operations environments.
Background in competency-based or mastery learning frameworks.
Experience building and analyzing assessments.
Teaching or similar experience in a high-accountability environment (Teach For America, urban education, or similar).

Responsibilities

Design and continuously improve assessments and certification frameworks that validly and reliably measure expert readiness for specific project types.
Build assessments and measurements of skills that are consistent, interpretable, and actually predictive of on-the-job performance.
Develop item banks, scoring guides, and inter-rater reliability protocols for evaluating complex human judgment tasks.
Run validity studies to ensure assessments measure intended constructs.
Analyze the relationship between instructional materials, assessments, and expert performance, identifying areas for improvement and making recommendations.
Analyze assessment data at the item level (difficulty, discrimination, reliability) and iterate based on findings.
Investigate the relationship between assessment performance and real-world expert effectiveness.
Build reports and dashboards to provide actionable insights to program and operations teams.
Design and analyze quasi-experimental, quantitative, and qualitative (mixed methods) studies to understand intervention impact on expert quality.
Track certification and assessment outcomes over time and identify programs needing revision.
Partner with learning designers and project teams to translate findings into program improvements.
Apply a continuous improvement mindset: ship, measure, learn, iterate.