Remote | Medicine Physician (MD/DO/Doctoral study/PhD)

24-Mag • New York, NY

2d • Remote

About The Position

We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario development, structured assessment, and expert review of model performance. Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical knowledge and reasoning, and help develop evaluation methods that reflect the complexity and nuance of evidence-based clinical practice.

Requirements

• Licensed physician currently engaged in active clinical practice
• MD, DO, or equivalent medical qualification
• Physicians from any medical specialty may be considered
• Strong experience with clinical decision-making
• Solid understanding of evidence-based medicine
• Ability to assess complex medical reasoning objectively
• Strong analytical and critical-thinking capabilities
• Excellent written and verbal communication skills
• Comfortable explaining nuanced clinical decisions clearly
• Ability to identify gaps in medical knowledge or reasoning
• Interest in how AI can support clinical practice
• Comfortable contributing to structured model-evaluation workflows
• Strong problem-solving abilities
• Able to collaborate effectively with research teams
• Comfortable working independently in a remote environment
• Able to integrate project work around existing clinical commitments
• Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party

Responsibilities

• Evaluate AI performance on realistic medical problems
• Assess the quality of clinical reasoning and decision-making
• Identify gaps in medical knowledge or clinical judgement
• Review model conclusions against evidence-based practice
• Distinguish significant clinical errors from minor limitations
• Create realistic scenarios for medical AI evaluation
• Design cases requiring nuanced clinical judgement
• Incorporate diagnostic and management decision points
• Develop scenarios reflecting real-world clinical complexity
• Ensure cases meaningfully test model reasoning capabilities
• Design systematic frameworks for medical AI assessment
• Define criteria for evaluating clinical reasoning quality
• Develop methods that capture nuance in clinical practice
• Support consistent assessment across repeated evaluation tasks
• Refine evaluation methods based on observed model performance
• Assess model understanding of clinical concepts
• Evaluate diagnostic and treatment-related reasoning
• Identify unsupported assumptions or missing considerations
• Apply specialty-specific expertise where relevant
• Provide structured rationale for evaluation decisions
• Collaborate with research teams on medical AI evaluation
• Communicate clinical findings clearly and concisely
• Provide actionable feedback on model strengths and limitations
• Contribute to strategies for improving medical model performance
• Support development of clinically meaningful benchmarks

What This Job Offers

• Job Type: Part-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree