Internal Medicine AI Evaluation Specialist

Weekday AI

2d•$130 - $180•Remote

About The Position

Join a leading AI research initiative focused on advancing healthcare-focused artificial intelligence. We are seeking experienced Internal Medicine physicians to contribute their clinical expertise toward training and evaluating next-generation AI models capable of sophisticated medical reasoning. In this role, you will develop realistic clinical scenarios, evaluate AI-generated responses against evidence-based standards, and help improve how AI systems reason about diagnosis, treatment planning, and patient management within Internal Medicine. We welcome Internal Medicine attending physicians, hospitalists, fellows, final-year residents, and other qualified Internal Medicine professionals who are passionate about applying their expertise to the future of healthcare AI.

Requirements

Attending Physicians: Board-certified in Internal Medicine. Current, active, and unrestricted medical license.
Resident Physicians: Final-year Internal Medicine resident. Recent residency graduates must be board-eligible.
Fellows: Board-certified or board-eligible in Internal Medicine or a related subspecialty. Current, active, and unrestricted medical license.
Professionals with expertise in Internal Medicine subspecialties are encouraged to apply, including: Cardiology, Hematology & Oncology, Endocrinology, Nephrology, Gastroenterology, Pulmonology, Other Internal Medicine subspecialties.

Nice To Haves

Strong knowledge of evidence-based medicine and clinical practice guidelines.
Experience in clinical teaching, medical research, or quality improvement initiatives.
Excellent written communication and analytical skills.
Interest in artificial intelligence, clinical decision support, or healthcare technology.

Responsibilities

Create realistic Internal Medicine case studies based on everyday clinical practice.
Develop prompts covering: Diagnostic reasoning, Differential diagnosis, Treatment planning, Risk stratification, Evidence-based guideline adherence, Long-term patient management.
Produce attending physician-level reference solutions that reflect current clinical best practices.
Ensure recommendations align with established evidence-based guidelines.
Review and assess AI-generated clinical responses using structured evaluation frameworks.
Identify reasoning gaps, inaccuracies, omissions, and opportunities for improvement.
Provide detailed written feedback to improve AI model performance and clinical reliability.
Participate in onboarding sessions and specialty calibration meetings.
Work with multidisciplinary teams to ensure consistency, accuracy, and quality across medical evaluations.