Remote | AI Safety Evaluation Specialist — $55–$65/hour

24-Mag•New York, NY

2d•Remote

About The Position

This is a specialized part-time consulting opportunity for experienced AI safety, trust and safety, public policy, journalism, scientific research, security, and content-evaluation professionals with strong judgment across complex and policy-sensitive subject matter. This role supports a frontier AI initiative focused on evaluating the safety, quality, factual accuracy, and alignment of advanced models. Selected professionals will review AI-generated responses across sensitive and ambiguous scenarios, apply structured safety policies and rubrics, identify behavioral failures, and provide detailed feedback that supports safer and more reliable model performance.

Requirements

At least 5 years of professional experience in AI safety, trust and safety, journalism, public policy, scientific research, security, content integrity, or a related field.
Strong analytical reasoning and the ability to assess nuanced, policy-sensitive scenarios consistently.
Excellent written English and the ability to explain complex evaluation decisions clearly.
Experience reviewing sensitive, high-risk, or ambiguous content.
Strong attention to factual accuracy, context, and policy interpretation.
Ability to work independently while applying detailed evaluation standards.
Professional residence in one of the eligible countries listed below.
A bachelor's degree or higher in journalism, communications, psychology, sociology, public policy, law, biology, chemistry, computer science, or a related discipline is highly relevant.
Equivalent specialist experience in safety evaluation, content integrity, scientific review, or risk analysis may also be considered.
Relevant research, policy, moderation, or AI evaluation work may strengthen an application.

Nice To Haves

Experience with AI safety, reinforcement learning from human feedback, supervised fine-tuning, trust and safety, or model evaluation.
Familiarity with content policies, safety standards, moderation frameworks, or rubric development.
Experience evaluating frontier AI models or language-model outputs.
Background in misinformation, political content, cybersecurity, biosecurity, scientific safety, or behavioral risk.
Experience participating in reviewer calibration or quality-assurance programs.
Familiarity with structured annotation, safety benchmarking, or human-feedback workflows.
Previous collaboration with researchers, engineers, policy specialists, or safety teams.
Graduate-level education in policy, behavioral science, security, law, life sciences, or artificial intelligence may be valuable.

Responsibilities

Evaluate AI-generated responses for safety, factual accuracy, policy compliance, relevance, and overall quality.
Assess whether outputs demonstrate appropriate judgment across nuanced and ambiguous scenarios.
Identify unsafe, misleading, incomplete, or poorly reasoned responses.
Compare alternative outputs and determine which response better satisfies safety and quality standards.
Review content involving misinformation, political persuasion, self-harm, violence, cybersecurity, biosecurity, fraud, and other sensitive areas.
Apply appropriate evaluation standards across high-risk and grey-area scenarios.
Distinguish between legitimate informational requests, potentially harmful content, and clear policy violations.
Evaluate whether model responses remain useful while handling sensitive subject matter responsibly.
Apply structured rubrics used in AI safety benchmarking, RLHF, and supervised fine-tuning workflows.
Assess outputs against defined criteria covering safety, accuracy, reasoning, and instruction adherence.
Identify ambiguity, gaps, or inconsistencies within evaluation guidelines.
Contribute to the refinement of scoring standards, policy interpretations, and reviewer instructions.
Identify hallucinations, unsafe outputs, reasoning failures, and policy-compliance issues.
Classify recurring model weaknesses and behavioral patterns.
Provide clear written explanations supporting each evaluation decision.
Collaborate with researchers and safety teams on calibration and ongoing evaluation initiatives.