STEM & Technical Specialist – Rubric Design

Xperteez Technology • ,

5d • Remote

About The Position

This is an hourly, remote contractor role for a STEM and technical subject matter expert. The specialist creates challenging, expert-level questions; evaluates AI-generated technical content; and designs detailed rubrics that define the requirements for correct, rigorous, expert-quality answers.

The role focuses on questions that test advanced reasoning rather than simple factual recall, and on identifying weaknesses in advanced AI models, including incorrect assumptions, incomplete reasoning, mathematical or scientific errors, missed edge cases, and plausible answers that fail expert scrutiny. The specialist translates complex technical judgment into clear, measurable evaluation criteria covering required concepts, reasoning steps, acceptable alternatives, critical errors, and partial-credit considerations. The role may also involve critiquing and improving rubrics written by other experts.

The position supports an AI Data Services company and subsidiary of SuperAnnotate that delivers training data for large AI companies and foundation-model labs. The specialist’s technical expertise helps improve AI model reasoning through the creation and evaluation of rigorous training and assessment content.

Requirements

• Bachelor’s degree or higher in Mathematics, Physics, Chemistry, Engineering, Computer Science, Statistics, Applied Science, or another technical discipline.
• Strong professional proficiency in English at minimum C1, with the ability to write precise technical explanations and evaluation criteria.
• At least 3 years of professional, academic, research, or industry experience in the stated technical domain.
• Demonstrated advanced expertise in a clearly defined technical field or specialization.
• Ability to create challenging technical questions requiring multi-step reasoning, domain expertise, or professional judgment.
• Ability to design detailed evaluation rubrics covering required reasoning, correct methodology, acceptable alternatives, partial-credit criteria, and critical errors.
• Ability to distinguish a correct final answer from an answer supported by valid versus flawed reasoning.
• Comfort reviewing and critiquing rubrics for ambiguity, missing criteria, redundancy, technical inaccuracies, or poor scoring design.
• High attention to detail when evaluating formulas, assumptions, units, methodology, edge cases, logical consistency, and technical terminology.
• Reliable and self-directed, with the ability to deliver consistent quality in an hourly, remote contractor workflow.

Nice To Haves

• Experience with exam writing, academic grading, peer review, research review, technical QA, standards development, or assessment design.
• Prior experience with AI evaluation, RLHF, SFT, benchmarking, data annotation, prompt design, or LLM evaluation.

Responsibilities

• Create difficult expert-level technical questions that test genuine reasoning and expose weaknesses in advanced AI systems.
• Write structured evaluation rubrics defining the criteria for correct, rigorous, and complete responses.
• Define partial-credit standards, including essential reasoning steps, acceptable alternatives, minor errors, and critical failures.
• Evaluate AI-generated solutions for correctness, reasoning quality, completeness, methodology, and technical precision.
• Identify hidden reasoning errors, including answers that reach a correct conclusion through invalid assumptions or flawed logic.
• Review other experts’ rubrics for technical accuracy, clarity, completeness, and scoring consistency.
• Produce expert answers, explanations, critiques, and other gold-standard reference content.
• Create questions, rubrics, evaluations, preference judgments, and reference-answer pairs for RL and SFT workflows.
• Ensure content reflects appropriate professional or academic standards in the relevant discipline.

What This Job Offers

• Career Level: Mid Level