Medical Expert AI Evaluation Specialist

Weekday AI

2d•$130 - $180•Remote

About The Position

This role is for one of our clients. Join a leading AI research initiative focused on advancing healthcare-focused artificial intelligence. We are seeking experienced medical professionals to contribute their clinical expertise toward training and evaluating next-generation AI models capable of sophisticated medical reasoning. In this role, you will help create realistic clinical scenarios, evaluate AI-generated responses using evidence-based medical standards, and provide expert insights that improve how AI systems reason about diagnosis, treatment, and patient care. This opportunity is ideal for physicians across a broad range of medical specialties who are passionate about applying their expertise to the future of healthcare technology.

Requirements

Attending Physicians: Board-certified physician with a current, active, and unrestricted medical license.
Resident Physicians: Currently in the final year of an accredited residency program. Recent residency graduates must be board-eligible.
Fellows: Board-certified or board-eligible in their primary specialty. Current, active, and unrestricted medical license.
Physicians from a wide range of specialties are encouraged to apply, including but not limited to: Family Medicine, Pediatrics, Anesthesiology, General Surgery, ENT (Otolaryngology), Physical Medicine & Rehabilitation (PM&R), Occupational Medicine, Other medical and surgical specialties.

Nice To Haves

Strong understanding of evidence-based medicine and clinical practice guidelines.
Excellent written communication skills.
Experience with medical education, clinical research, peer review, or healthcare quality initiatives.
Interest in artificial intelligence and healthcare innovation.

Responsibilities

Create realistic, specialty-specific clinical cases that reflect real-world medical practice.
Design prompts covering areas such as: Diagnostic reasoning, Treatment planning, Risk stratification, Differential diagnosis, Guideline-based clinical decision-making.
Produce high-quality clinical responses that reflect attending physician-level reasoning.
Ensure recommendations align with current evidence-based practices and accepted clinical guidelines.
Assess AI-generated medical responses using structured evaluation frameworks and scoring rubrics.
Identify strengths, weaknesses, omissions, and reasoning errors.
Provide detailed written feedback to improve model performance and clinical accuracy.
Participate in onboarding sessions and specialty calibration meetings.
Work with multidisciplinary research teams to maintain consistency and quality across medical evaluations.