Physician (MD/DO) - Medical AI Evaluation

Mindrift•New York, NY

1d•Remote

About The Position

Mindrift is recruiting actively practicing physicians to evaluate clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the main focus. The project is intended for US-licensed physicians seeking flexible, time-limited work that can fit around clinical practice. Work involves reviewing written clinical cases only; there is no patient contact or malpractice exposure. Participation is project-based, and contributors can work from anywhere and choose when and how much to contribute while the project is active. Tasks are estimated at approximately 10–20 hours per week during active project phases. This is an estimate, not a minimum or guaranteed workload. Tasks must be completed within the project timeline and meet the stated acceptance criteria to be approved. Contributors are paid a flat rate per approved task, set at about a $100-per-hour equivalent based on estimated task time. QA and adjudication tasks are paid at a higher rate, and high-quality approved tasks may earn additional incentives. A one-time $50 onboarding reward is paid after the first production task is approved. The project workflow is: apply, pass qualification(s), join a project, complete tasks, and get paid. Reviewers evaluate clinical vignettes paired with questions, proposed answers, and distractor “trap” answers; assess benchmark-item quality and clinical reasoning; and write short rationales. Work is conducted through three independent blind reads followed by physician adjudication.

Requirements

Medical degree (MD or DO).
Active, unrestricted US medical license, verified at onboarding and re-checked monthly.
Board certification or completed residency training, with recent direct patient care; attending-level experience is preferred.
Broad, multi-system diagnostic and treatment experience, such as in internal, family, emergency, or hospital medicine.
Strong clinical reasoning skills, including differential diagnosis, next-step management, and application of evidence-based guidelines.
Strong written English proficiency (C1+).

Nice To Haves

Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/QA.

Responsibilities

Evaluate clinical EHR vignettes paired with questions, proposed answers, and distractor “trap” answers across diagnosis and treatment tasks and multiple body systems.
Score benchmark items for vignette accuracy and completeness, whether the vignette gives away the answer, answer correctness and gradeability, and trap quality.
Assess whether the reasoning chain reaches the answer using vignette facts alone, evaluate reasoning traces, and write short rationales.
Participate in a review process involving three independent blind reads followed by physician adjudication.
Submit tasks within project timelines and meet the listed acceptance criteria.