Physician (MD/DO) - Medical AI Evaluation

Mindrift • New York, NY

18h

About The Position

Mindrift connects specialists with project-based AI opportunities for leading technology companies to test, evaluate, and improve AI systems. This opportunity is for US-based, actively practicing physicians to evaluate clinical EHR vignettes for a medical AI evaluation program, with a primary focus on general and internal medicine. Projects are non-permanent and project-based.

Depending on the project, contributors evaluate clinical vignettes paired with questions, proposed answers, and distractor “trap” answers across multiple body systems. They assess vignette accuracy and completeness, whether a vignette reveals its answer, answer correctness and gradeability, and trap quality. They also review whether reasoning follows from the vignette facts, assess reasoning traces, and write short rationales. Work involves three independent blind reads followed by physician adjudication.

Tasks are estimated to take around 10–20 hours per week during active project phases; this is not a guaranteed workload and applies only while a project is active. Tasks must meet acceptance criteria and be submitted by the deadline. Compensation is per accepted task, with rates depending on qualification tier and task completion efficiency, up to the equivalent of $70 per hour. Applicants should submit a CV in English and indicate their English proficiency.

Requirements

• Medical degree (MD or DO).
• Active, unrestricted US medical license, verified with the issuing medical board.
• Board certification or completed residency training, with recent direct patient care; attending-level experience is preferred.
• Broad, multi-system diagnostic and treatment experience in internal, family, emergency, or hospital medicine.
• Strong clinical reasoning, including differential diagnosis, next-step management, and application of evidence-based guidelines.
• Strong written English proficiency (C1+).
• Must be a US-based, actively practicing physician.
• Submit a CV in English and indicate English proficiency.

Nice To Haves

• Prior experience in medical AI evaluation, clinical content or exam-question review, or medical annotation/quality assurance.
• Attending-level experience.

Responsibilities

• Evaluate clinical EHR vignettes paired with questions, proposed answers, and distractor trap answers across multiple medical systems.
• Score benchmark items for vignette accuracy and completeness, answer disclosure, answer correctness and gradeability, and trap quality.
• Assess whether the reasoning chain reaches the answer from vignette facts alone.
• Review reasoning traces and write short rationales.
• Participate in three independent blind reads followed by physician adjudication.
• Submit tasks by the deadline and meet the stated acceptance criteria.

What This Job Offers

• Job Type: Part-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree