Sr. Psychometrician, Data Center Certifications

Amazon • Herndon, VA

26d

About The Position

We're looking for an experienced Psychometrician to serve as the measurement
science backbone of our certification programs. In this role, you'll ensure that
every assessment we build is valid, reliable, fair, and defensible — at global
scale. You'll work at the intersection of workforce capability, assessment
design, and data-driven quality assurance, partnering closely with subject
matter experts, instructional designers, and program managers to translate
complex technical competencies into rigorous, scalable assessments.

n your first 90 days, you'll complete a psychometric audit of existing
assessments, establish quality benchmarks, and begin embedding rigorous
measurement practices into our item development and review workflows. Within six
months, you'll be the go-to expert on assessment validity and reliability across
the program — and your fingerprints will be on every certification we launch.

This is a high-impact individual contributor role with significant influence
over program strategy and quality standards.

Requirements

• Knowledge of data analysis languages (e.g., R, Stata, Python, etc.) or other statistical software
• Master's degree in Psychometrics, Educational Measurement, Industrial-Organizational Psychology, or a related field
• 5+ years of experience in psychometrics, assessment development, or credentialing programs
• Demonstrated experience with JTA facilitation, item development, and standard-setting methodologies
• Experience working with large-scale, high-stakes certification or licensure programs

Nice To Haves

• Ph.D. in econometrics, statistics, industrial engineering, operations research, optimization, data mining, analytics, or equivalent quantitative field
• Experience in technical or workforce credentialing programs (e.g., trades, engineering, operations)
• Familiarity with performance-based and simulation-based assessment design
• Experience managing vendor relationships in an assessment or credentialing context
• Ability to operate effectively in a fast-paced, globally distributed environment

Responsibilities

• Lead the psychometric design and validation of certification assessments across role-based and specialty-based programs, including written, practical, and performance-based formats
• Design and audit scalable Job Task Analyses (JTAs) in partnership with subject matter experts and operations stakeholders to establish defensible content blueprints
• Apply classical test theory (CTT) and item response theory (IRT) to evaluate item and test quality; conduct standard-setting studies (e.g., Angoff, Bookmark) to establish and validate cut scores
• Develop and govern item development, review, and validation workflows across global content contributors, ensuring consistency and rigor
• Analyze assessment data to identify bias, differential item functioning (DIF), and reliability issues; recommend and implement remediation strategies
• Maintain independence between training and certification development to protect assessment integrity
• Build and maintain psychometric reporting frameworks that surface actionable insights on pass rates, item performance, and certification quality
• Partner with third-party assessment vendors on test development, delivery, and quality assurance; hold vendors accountable to psychometric standards
• Communicate complex psychometric concepts clearly to non-technical stakeholders, including senior leadership

Benefits

• health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage)
• 401(k) matching
• paid time off
• parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level