Domain Expert, AI Evaluation Task Writer: Healthcare (Contract, Remote)

Patronus AI, Inc.

5d • $50 - $150 • Remote

About The Position

Patronus AI is hiring healthcare domain experts as independent contractors to create realistic evaluation tasks for AI models. Based on provided scenarios, contractors build fully fictional patient or operational records, write task requests, solve each task, document reference solutions, and define explicit scoring criteria. Tasks should be grounded in professional healthcare knowledge and designed to reveal where current AI models fail. Contractors also test tasks against current models, refine their difficulty, review their own and peers’ work for accuracy and ambiguity, incorporate reviewer feedback, and flag guideline gaps or edge cases. All task content must be human-authored and original; real patient data and PHI may not be used.

The role is fully remote and asynchronous, with a commitment of at least 10 hours per week. Work is paid at $50–$150 per hour for task authoring, with payment upon submission of the full task and a potential bonus for each accepted task. Strong performers may move higher within the pay range or to productivity-based pay. The paid trial task is followed by onboarding and an initial project. Work begins on a rolling basis after passing the trial task.

Requirements

• At least 3 years of hands-on experience in one or more of the listed clinical practice, clinical documentation and coding, utilization management and care coordination, or revenue cycle and payer operations areas.
• Current or past licensure or certification in the relevant field; examples include MD/DO, NP, PA, RN, PharmD, CPC, CCS, CDIP, CCDS, CCM, or CRCR.
• Working knowledge of the guidelines, criteria, and documentation standards governing the relevant area, such as clinical practice guidelines, ICD-10-CM/PCS and CPT coding rules, InterQual or MCG criteria, and CMS or payer policies.
• Careful reading and the habit of citing the chart entry, code set, or policy supporting each conclusion.
• Clear, precise writing in professional English.
• Ability to work independently and asynchronously against deadlines.

Nice To Haves

• Board certification or credentials beyond the minimum for the role, such as a clinical specialty board, CPHQ, CRCR, or CHFP.
• Experience across settings such as inpatient, outpatient, payer, pharmacy, or post-acute.
• Experience with relevant systems, such as Epic or Oracle Health, Solventum 360 Encompass, InterQual or MCG, Availity or a payer portal, and a claims clearinghouse.
• Teaching, precepting, board-question writing, clinical case development, or prior experience writing or reviewing AI evaluation tasks.
• Familiarity with AI model failure modes, such as fabricated citations, outdated guidelines, missed contraindications, incorrect codes, or overconfident answers.

Responsibilities

• Build realistic, fully fictional patient and operational records from provided scenarios, including clinical records, coding materials, utilization review documents, claims, payer materials, and related records.
• Write clear task requests that specify the deliverable and explain how uncertainty and justified assumptions should be handled.
• Solve tasks end to end and document reference solutions, including the reasoning behind each step.
• Write explicit scoring criteria that support consistent grading by independent graders, including acceptable alternatives and unsafe responses.
• Run tasks against current AI models, identify how and why they fail, and adjust task difficulty.
• Review personal and peer tasks for clinical accuracy, ambiguity, exposed answers, and shortcuts that bypass the intended reasoning.
• Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases.

What This Job Offers

• Career Level: Mid Level
• Education Level: No Education Listed