Physician Reviewer (MD/DO) - Medical AI Evaluation

Mindrift•Provo, UT

1d•Remote

About The Position

Mindrift is seeking actively practicing physicians with an active, unrestricted US medical license to evaluate clinical EHR vignettes for a medical AI evaluation project. The main focus is general and internal medicine. This is a project-based opportunity designed to fit around clinical practice, with no patient contact or malpractice exposure. Contributors review written clinical cases and can choose when and how much to work. Tasks are estimated at about 10–20 hours per week during active project phases; this is not a minimum or guaranteed workload. Work follows three independent blind reads, followed by physician adjudication. Tasks must meet the project’s acceptance criteria to be approved. Contributors assess clinical EHR vignettes paired with a question, proposed answer, and distractor “trap” answer across diagnosis and treatment topics and multiple organ systems. They evaluate the quality of benchmark items, assess whether clinical reasoning follows from the vignette, review reasoning traces, and write short rationales. Compensation is per approved task at a flat rate estimated to equal about $100 per hour based on estimated task time. QA and adjudication tasks are paid at a higher rate. A one-time $50 onboarding reward is paid after the first production task is approved, and additional incentives may be available for high-quality approved tasks under terms provided in project guidelines.

Requirements

Medical degree (MD or DO).
Active, unrestricted US medical license, verified at onboarding and re-checked monthly.
Board certification or completed residency training.
Recent direct patient care; attending-level experience preferred.
Broad, multi-system diagnostic and treatment experience in internal, family, emergency, or hospital medicine.
Strong clinical reasoning, including differential diagnosis, next-step management, and application of evidence-based guidelines.
Strong written English (C1+).

Nice To Haves

Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/QA.

Responsibilities

Evaluate clinical EHR vignettes paired with a question, proposed answer, and distractor trap answer across diagnosis and treatment tasks.
Assess cases spanning cardiovascular, nervous, hematologic, respiratory, digestive, urinary, reproductive, musculoskeletal, and integumentary systems.
Score benchmark-item quality, including vignette accuracy and completeness, whether the vignette gives away the answer, answer correctness and gradeability, and trap quality.
Check whether the reasoning chain reaches the answer using only facts from the vignette, assess reasoning traces, and write short rationales.
Participate in a workflow involving three independent blind reads followed by physician adjudication.
Submit tasks within project timelines and meet the listed acceptance criteria.