Remote | Resident Medical Specialist (MD/DO)

24-Mag • New York, NY

2d • Remote

About The Position

We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario design, benchmark development, and structured assessment of model performance. Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical reasoning, and help develop assessment methods that reflect the complexity and nuance of evidence-based practice. The opportunity is open across medical specialties.

Requirements

• Licensed physician currently engaged in active clinical practice
• MD, DO, or equivalent medical qualification
• Open to physicians across medical specialties
• Strong clinical decision-making experience
• Solid understanding of evidence-based medicine
• Ability to assess complex medical reasoning objectively
• Strong analytical and critical-thinking capabilities
• Excellent written and verbal communication skills
• Comfortable explaining nuanced clinical decisions clearly
• Able to identify gaps in medical knowledge or reasoning
• Interest in the application of AI to clinical practice
• Comfortable collaborating with research teams in a remote environment
• Able to work independently around existing clinical commitments
• Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party

Nice To Haves

• Relevant backgrounds may include Public Health, Pediatrics, Emergency Medicine, Obstetrics & Gynecology, or Gynecological Oncology

Responsibilities

• Evaluate AI performance on realistic medical reasoning tasks
• Assess whether clinical conclusions are appropriately supported
• Identify gaps in medical knowledge or decision-making
• Review model reasoning against evidence-based practice
• Distinguish clinically significant errors from minor limitations
• Create realistic medical scenarios for model evaluation
• Design cases that require nuanced clinical judgement
• Incorporate diagnostic and management decision points
• Develop scenarios reflecting real-world clinical complexity
• Ensure cases meaningfully test model capabilities
• Design systematic assessment frameworks for medical AI systems
• Define criteria for evaluating clinical reasoning quality
• Develop methods that capture nuance in medical decision-making
• Support consistent evaluation across repeated clinical tasks
• Refine assessment approaches based on observed model performance
• Identify areas where models demonstrate weak clinical understanding
• Evaluate diagnostic reasoning and treatment-related judgement
• Assess model responses for clinical relevance and completeness
• Apply specialty-specific expertise where appropriate
• Provide structured rationale for evaluation decisions
• Collaborate with research teams on medical AI evaluation
• Communicate clinical findings clearly and concisely
• Provide actionable feedback on model strengths and limitations
• Contribute to strategies for improving medical model performance
• Support development of clinically meaningful benchmarks

What This Job Offers

• Job Type: Part-time
• Career Level: Mid Level
• Education Level: Ph.D. or professional degree