AI Policy Generalist - Remote US

Handshake•Remote, Remote

5d•$45 - $55•Remote

About The Position

As an AI Policy Generalist, you will turn complex customer policies into consistent, well-reasoned evaluations of AI model behavior. You will read user requests, model responses, and relevant conversation history, then determine which policy category best applies. The most interesting cases will not have obvious answers. Two examples may look almost identical until a single word, contextual detail, or difference in intent changes the correct classification. We are looking for people who enjoy splitting hairs in a healthy way. You form clear opinions, explain precisely why two cases should be treated differently, challenge interpretations respectfully, and change your mind when better evidence emerges. You understand that productive disagreement is not about winning an argument. It is how a team finds the most accurate and consistent interpretation. This is not rote annotation. Policies cannot anticipate every possible edge case, and good evaluators do not apply them mechanically. You will balance the policy’s text and intent with customer expectations, conversation context, precedent, and team calibration. The subject matter will vary. One project may involve distinguishing benign assistance from meaningful facilitation of harm. Another may require evaluating whether an interaction reflects ordinary emotional support or unhealthy reliance. A third may focus on nuanced boundaries within sexual-safety policy. Success requires learning each customer’s framework on its own terms rather than carrying assumptions from one domain into another.

Requirements

Enjoy making precise distinctions between cases that other people might consider equivalent
Notice when one word, contextual detail, or change in intent materially affects the answer
Can hold a strong opinion without becoming attached to being right
Explain judgment calls clearly enough that another person can audit your reasoning
Ask productive questions when a policy is ambiguous instead of guessing or forcing certainty
Can separate your personal views from the standard a customer has asked you to apply
Comfortable discussing disagreement directly, respectfully, and without making it personal
Can follow the letter of a policy while also understanding its purpose and underlying logic
Remain careful and consistent during repetitive, feedback-heavy work
Learn unfamiliar subject matter quickly and know when additional context is needed
Intellectually curious, self-directed, and comfortable working in a fast-changing environment
Communicate clearly and precisely in writing
Treat sensitive information and difficult subject matter with maturity and sound judgment
Strong candidates may come from quality assurance, research, editing, law, teaching, operations, trust and safety, content moderation, social science, policy, investigations, compliance, customer support, or other fields that require careful interpretation and defensible decision-making.
Care more about how you reason than where you learned to reason.

Nice To Haves

Experience evaluating or comparing outputs from ChatGPT, Claude, Gemini, or other language models in a professional capacity
Prior work in AI evaluation, data annotation, RLHF, model quality, trust and safety, policy operations, or content moderation
Experience applying detailed rubrics, taxonomies, regulatory language, editorial standards, or quality frameworks
Familiarity with calibration sessions, inter-rater agreement, quality audits, or adjudication workflows
Experience writing policy guidance, decision trees, evaluation examples, or structured rationales
Comfort working with long conversations, incomplete context, and conflicting evidence
Familiarity with AI safety, responsible AI, or the ways language models can assist, mislead, or cause harm
Prior AI evaluation experience is helpful, but it is not required.

Responsibilities

Learn new customer policies, definitions, taxonomies, and evaluation rubrics quickly
Evaluate user requests and AI model responses within the full relevant conversation context
Distinguish between closely related labels, severity levels, and policy boundaries
Select the most defensible classification when a case is genuinely ambiguous
Write concise, evidence-based rationales that cite relevant policy language and conversation details
Identify policy gaps, contradictions, unclear definitions, and emerging edge cases
Raise thoughtful questions when existing guidance does not resolve a case
Participate actively in calibration discussions with evaluators, project leads, policy teams, and researchers
Challenge interpretations respectfully and update your judgment when new guidance or stronger reasoning emerges
Apply customer policy consistently without substituting personal beliefs for the policy standard
Maintain accuracy and attention to detail across repeated evaluations
Incorporate feedback quickly and apply clarified guidance to future work
Help improve evaluation frameworks, examples, decision rules, and quality standards
Move effectively between projects covering different policy domains and customer needs