Remote | AI Evaluation Guidelines & Rubric Specialist — $40–$60/hour

24-Mag•New York, NY

2d•Remote

About The Position

This is a specialized full-time consulting opportunity for US-based linguists, instructional designers, and technical writers experienced in developing clear evaluation guidelines, structured rubrics, and human-rating instructions for generative AI programmes. This role supports a high-impact generative AI initiative focused on translating complex and potentially ambiguous programme requirements into precise, practical guidance for human evaluators. Selected professionals will develop rater-ready instructions across domains such as finance, retail, insurance, legal, and sports while resolving contradictions, defining edge cases, and improving consistency throughout evaluation workflows.

Requirements

At least 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field.
Direct experience developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programmes.
Demonstrated ability to resolve ambiguity and contradiction in complex written specifications.
Experience translating specialist requirements into clear and practical instructions.
Ability to work effectively across multiple subject-matter domains.
A portfolio or concrete examples showing measurable improvements to guidelines, rubrics, or instructional materials.
Demonstrable professional growth and increasing responsibility.
Reliable availability for at least 35 hours per week during weekdays.
A degree in linguistics, instructional design, education, communications, technical writing, language studies, or a related field is highly relevant.
Direct experience developing rater guidelines or rubrics for generative AI or RLHF programmes is required.
Applicants should be prepared to provide concrete examples of guideline or specification improvements.
Immediate availability is preferred.

Nice To Haves

Graduate-level education in applied linguistics, learning design, human-computer interaction, or information design may be helpful.
Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be considered.
Training in assessment design, taxonomy development, content strategy, or quality assurance may be valuable.
Experience supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programmes.
Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes.
Experience developing domain-specific guidance for finance, insurance, retail, legal, sports, or comparable fields.
Knowledge of controlled language, information architecture, taxonomy design, or content governance.
Experience conducting guideline usability tests or analysing inter-rater consistency.
Familiarity with version control, documentation systems, and structured authoring tools.
Previous collaboration with researchers, programme managers, engineers, and subject matter experts.

Responsibilities

Translate programme requirements into clear, structured, and actionable instructions for human evaluators.
Develop guidelines that can be applied consistently across standard scenarios and complex edge cases.
Define terminology, rating criteria, decision rules, exceptions, and escalation pathways.
Ensure instructions are accessible to raters while preserving necessary domain-specific precision.
Design detailed scoring rubrics for evaluating generative AI outputs.
Establish measurable criteria covering correctness, relevance, reasoning quality, completeness, and instruction adherence.
Create examples and counterexamples illustrating different performance levels.
Align evaluation frameworks with programme objectives and quality standards.
Review draft specifications for ambiguity, contradiction, missing information, and inconsistent terminology.
Identify instructions that may lead to conflicting interpretations across raters.
Revise guideline sets until they can be applied reliably with minimal escalation.
Document concrete before-and-after improvements to written requirements and evaluation instructions.
Convert specifications from finance, retail, insurance, legal, sports, and other specialist domains into rater-ready guidance.
Collaborate with subject matter experts to understand domain-specific terminology and professional judgment.
Preserve important technical nuance while making instructions clear to non-specialist evaluators.
Maintain consistent structure and quality across multiple domain-specific guideline sets.