Research Scientist II - AMZ9698004
About The Position
Perform and support the main psychometric aspects of exam development and operations, including but not limited to automated test assembly, item and test analyses, optimal item bank design, job task analysis, standard setting, quality assurance, and project planning. Conduct main aspects of psychometric analysis in operational work including performing item analysis using psychometric methods, building optimal test forms and pools via optimization techniques, analyzing and monitoring item bank health, setting pass standards via standard setting studies, and supporting Job Task Analysis (JTA) to define and refresh test blueprints. Conduct main aspects of psychometric analysis in developing and applying statistical and psychometric modeling to evaluate and ensure AWS certification exams’ validity, reliability, applicability, efficiency, and accuracy. Participate in research projects to improve existing operational processes and quality using advanced techniques such as Machine Learning (ML), statistical modeling, Natural Language Processing (NLP), Generative Artificial Intelligence (GenAI), etc. Develop automation code using R or Python for psychometric workflow pipeline and other tasks to improve operational efficiencies. Present, interpret, and communicate the results of analyses to stakeholders through written and oral reports. Follow the accreditation standards set by ISO/IEC:2012 17024 and the National Council for Certifying Agencies (NCCA) as they relate to valid psychometric practices. Engage with the professional community through conferences and publications.
Requirements
- PhD or foreign equivalent degree in Statistics, Psychometrics, Educational Measurement, Quantitative Psychology, Data Science, Industrial-Organizational (I/O) Psychology, or a related field and one year of research or work experience in the job offered, or as a Research Scientist, Research Assistant, Software Engineer, or a related occupation.
- Must have 1 year of experience in large-scale education, licensure, or certification assessment programs.
- Must have 1 year of experience in operational psychometric tasks on large-scale education, licensure, or certification assessment programs including item analysis, equating and scaling, item response theory, classical test theory, form and pool assembly, item bank health analysis, standard setting, and job task analysis.
- Must have 1 year of experience in at least one of the complex test designs such as linear-on-the-fly testing (LOFT), computerized adaptive testing (CAT).
- Must have 1 year of experience in at least one of the following areas including machine learning (ML) or natural language processing (NLP).
- Must have 1 year of experience in programming skills in at least one script-based programming language (R, Python).
Nice To Haves
- Please see job description and the position requirements above.
Responsibilities
- Perform and support the main psychometric aspects of exam development and operations.
- Conduct main aspects of psychometric analysis in operational work.
- Conduct main aspects of psychometric analysis in developing and applying statistical and psychometric modeling.
- Participate in research projects to improve existing operational processes and quality using advanced techniques such as Machine Learning (ML), statistical modeling, Natural Language Processing (NLP), Generative Artificial Intelligence (GenAI), etc.
- Develop automation code using R or Python for psychometric workflow pipeline and other tasks to improve operational efficiencies.
- Present, interpret, and communicate the results of analyses to stakeholders through written and oral reports.
- Follow the accreditation standards set by ISO/IEC:2012 17024 and the National Council for Certifying Agencies (NCCA) as they relate to valid psychometric practices.
- Engage with the professional community through conferences and publications.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
Ph.D. or professional degree
