Legal & Compliance Specialist – Rubric Design

Xperteez Technology • ,

8h • Remote

About The Position

This hourly, remote contractor role is for a Legal & Compliance Subject Matter Expert who creates challenging professional questions, evaluates AI-generated legal and regulatory content, and designs high-quality rubrics defining complete, well-reasoned, and professionally appropriate responses. Tasks may involve legal analysis, regulatory interpretation, compliance, issue spotting, risk assessment, policy interpretation, contracts, privacy, financial regulation, employment matters, corporate compliance, or related fields, depending on the specialist’s expertise.

The specialist evaluates whether AI-generated responses identify relevant issues, apply appropriate rules or standards, recognize exceptions, distinguish facts from assumptions, address relevant risks, and avoid unsupported conclusions. They translate nuanced legal or regulatory judgment into structured evaluation criteria addressing issue identification, applicable standards, analysis quality, completeness, exceptions, jurisdictional considerations, risk, and professional reasoning.

The role supports an AI Data Services company and subsidiary of SuperAnnotate that provides training data to AI companies and foundation-model labs. The specialist’s work helps improve the accuracy, rigor, and professional reliability of AI models’ legal and compliance outputs. Interview language is English.

Requirements

• JD, LLB, LLM, law degree, compliance qualification, or equivalent professional background relevant to the stated specialty.
• Strong professional proficiency in English, minimum C1.
• At least 3 years of professional experience in law, regulatory compliance, legal research, policy, risk, privacy, financial compliance, contracts, or a related field.
• Demonstrated expertise in a clearly defined legal, regulatory, or compliance domain.
• Ability to create challenging professional scenarios requiring issue spotting, interpretation, analysis, and judgment.
• Strong ability to design rubrics defining required issues, applicable standards, reasoning steps, relevant exceptions, risks, and critical omissions.
• Ability to distinguish legitimate alternative legal interpretations from objectively deficient analysis.
• Comfort reviewing and critiquing rubrics developed by other legal or compliance experts.
• Strong attention to detail when evaluating terminology, factual assumptions, jurisdiction, authority, exceptions, materiality, and conclusions.

Nice To Haves

• Professional legal qualification or bar admission is strongly preferred for attorney-focused roles.
• Experience with legal education, examination writing, peer review, legal editing, quality assurance, policy drafting, compliance testing, or supervisory review is strongly preferred.
• Prior experience with AI evaluation, legal AI, data annotation, RLHF, model benchmarking, or rubric-based scoring is preferred.

Responsibilities

• Develop realistic, difficult legal and compliance scenarios requiring expert analysis and professional judgment.
• Design detailed rubrics defining relevant issues, rules, standards, analytical steps, exceptions, risks, and critical errors.
• Evaluate AI-generated analysis for accuracy, completeness, reasoning quality, relevance, and professional rigor.
• Determine whether models identify material legal or regulatory considerations.
• Assess reasoning quality, including identifying unsupported conclusions, incorrect application of rules, missing exceptions, and faulty assumptions.
• Review peer rubrics for missing issues, ambiguity, redundancy, excessive subjectivity, or inaccurate legal expectations.
• Create high-quality analyses, answer frameworks, critiques, and gold-standard examples.
• Produce questions, rubrics, evaluations, preference data, and reference-answer pairs for RL and SFT workflows.
• Ensure content appropriately distinguishes factual analysis, legal interpretation, professional judgment, and uncertainty.

What This Job Offers

• Career Level: Mid Level