Domain Expert, AI Evaluation Task Writer: Legal (Contract, Remote)

Patronus AI, Inc.

5d • $50 - $150 • Remote

About The Position

Patronus AI is hiring a remote independent contractor to write legal evaluation tasks for AI models. The contractor will create realistic legal scenarios and working documents based on supplied scenarios, write task requests, complete each task and document a reasoned reference solution, and establish clear scoring criteria. Work may cover civil litigation, corporate and M&A transactions, commercial contracts, and privacy and data subject requests. The role also involves testing tasks against current AI models, checking tasks for legal accuracy and ambiguity, reviewing peers’ work, and incorporating reviewer feedback. The engagement is asynchronous and requires at least 10 hours per week. Pay is $50–$150 per hour for task authoring, paid upon full task submission, with a potential bonus for accepted tasks; the trial task is paid. Start is rolling after passing the trial task.

All task documents and solutions must be human-authored and original. Real client, matter, employer, or personal information, and information subject to privilege or confidentiality obligations, must not be used; all parties and scenarios must be fictional.

Requirements

• At least 3 years of hands-on experience in one or more of the listed legal areas, in a role such as paralegal, legal assistant, contracts manager or specialist, privacy or data protection analyst, or attorney; experience must include doing the work, not only supervising it.
• Working knowledge of the rules, procedures, and documentation standards governing the relevant area.
• Careful reading and a practice of citing the clause, message, or docket entry supporting each conclusion.
• Clear, precise writing in professional English.
• Ability to work independently and asynchronously against deadlines.
• Availability for at least 10 hours per week.
• Ability to produce original, human-authored materials without using real or confidential information.

Nice To Haves

• Paralegal certification, such as NALA CP or NFPA RP; a law degree; CIPP/US or CIPP/E certification; or a contract management credential.
• Experience with relevant systems, such as Relativity, Everlaw, DISCO, a virtual data room, Ironclad, Icertis, Agiloft, Conga, OneTrust, TrustArc, or Transcend.
• Law firm, in-house, and legal operations experience across more than one setting.
• Prior experience writing or reviewing AI evaluation tasks, exam questions, case studies, or training material.
• Familiarity with AI model failure modes, including fabricated citations, misread clauses, miscounted deadlines, and overconfident conclusions.

Responsibilities

• Build realistic evidence files from provided scenarios, including legal agreements, correspondence, pleadings, discovery materials, privilege logs, docket entries, closing materials, contract playbooks and redlines, and privacy request records.
• Write clear task requests specifying the deliverable and how uncertainty and justified assumptions should be handled.
• Solve each task end to end and document a reference solution with reasoning for every step.
• Write explicit scoring criteria that support consistent grading by independent graders, including acceptable alternative answers and limits on the solver’s authority.
• Run tasks against current AI models, identify where and why they fail, and adjust task difficulty.
• Review personal and peer tasks for legal accuracy, ambiguity, answers exposed in the materials, and shortcuts that avoid the intended reasoning.
• Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases.

What This Job Offers

• Education Level: No Education Listed