Physician (MD/DO) - Medical AI Evaluation

Mindrift

1d

About The Position

Mindrift connects specialists with project-based AI opportunities for leading technology companies focused on testing, evaluating, and improving AI systems. This opportunity is for US-based, actively practicing physicians to evaluate clinical EHR vignettes for a medical AI evaluation program, with a primary focus on general and internal medicine. Projects are non-permanent and participation is project-based.

Contributors review clinical vignettes paired with questions, proposed answers, and distractor answers across multiple body systems. They assess benchmark-item quality and clinical reasoning, verify whether answers and reasoning are supported by the vignette, and provide short rationales. The process includes three independent blind reads followed by physician adjudication.

Tasks are estimated to require around 10–20 hours per week during active project phases; this is not guaranteed and applies only while the project is active. Work must be submitted by the deadline and meet acceptance criteria. Compensation is per accepted task, with rates depending on qualification tier and task efficiency, up to the equivalent of $70 per hour.

Requirements

• Medical degree (MD or DO).
• Active, unrestricted US medical license, verified with the issuing medical board.
• Board certification or completed residency training, with recent direct patient care; attending-level experience is preferred.
• Broad, multi-system diagnostic and treatment experience in internal, family, emergency, or hospital medicine.
• Strong clinical reasoning, including differential diagnosis, next-step management, and application of evidence-based guidelines.
• Strong written English proficiency (C1+).
• Must be a US-based, actively practicing physician; the opportunity is not suitable for medical students, unlicensed medical graduates, non-physician clinicians, or non-clinical healthcare titles.

Nice To Haves

• Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/QA.
• Attending-level clinical experience.

Responsibilities

• Evaluate clinical EHR vignettes paired with questions, proposed answers, and distractor “trap” answers across diagnosis and treatment tasks.
• Score benchmark items for vignette accuracy and completeness, whether the vignette gives away the answer, answer correctness and gradeability, and trap quality.
• Check whether the reasoning chain reaches the answer using only facts from the vignette.
• Assess the correctness of reasoning traces and write short rationales.
• Participate in three independent blind reads followed by physician adjudication.
• Submit tasks by the deadline and meet the listed acceptance criteria.

What This Job Offers

• Job Type: Part-time
• Career Level: Mid Level
• Education Level: Ph.D. or professional degree