Senior AI/ML Engineer - Validation & Evaluation

Mayo Clinic • Rochester, MN

1d • $141,024 - $204,526

About The Position

Mayo Clinic is a renowned healthcare provider, recognized for its top-tier specialties and commitment to both patient care and employee well-being. We offer competitive compensation, comprehensive benefits, and continuous learning opportunities to foster long-term career growth. This role is within the AI Validation & Monitoring (AVM) team, focusing on providing practice leadership for validation pathways, applied evaluation, and consultation for AI/ML systems in a healthcare setting.

Requirements

• A master’s degree in engineering, computer science, mathematics, health science, or a related field with 4 years of experience, OR a bachelor’s degree with 6 years of experience.
• Extensive experience applying AI and machine learning in production healthcare environments or similar highly regulated or technology focused industries, showcasing an understanding of healthcare technology.
• Demonstrated leadership in managing complex projects, with a proven ability to navigate intricate project requirements and deliver successful outcomes.
• Proficiency in fostering collaboration across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
• Demonstrated expertise in cloud infrastructure environment and software development tools.
• Experience working with large, complex, and heterogeneous data sets, preferably in healthcare.
• Skilled in AI/ML techniques and frameworks.
• Familiarity with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
• Demonstrated initiative in administration, education, software development, and technical reporting.
• A commitment to mentoring and training less-experienced team members, coupled with strong interpersonal, communication, and time management skills.

Nice To Haves

• A Ph.D. or other doctorate is preferred.
• Strong expertise in AI/ML techniques and frameworks, such as deep learning, natural language processing, and Generative AI, with proficiency in tools like Python, TensorFlow, PyTorch, sci-kit-learn, Keras, etc.
• Knowledge of the healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
• Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
• Demonstrated experience leading technical/quantitative teams in a regulated environment.
• Demonstrated experience creating risk management files and verification/validation strategies for digital health technology products within the healthcare industry.
• Strong expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development.
• Ability to conduct expert reviews using established usability practices and methods.
• Presents findings in easy-to-understand terms for the business or clinical practice.
• Demonstrated hands-on experience using the TRex assessment application to lead validation pathway reviews for AI tools deployed in Epic, ANIMATE, or comparable clinical environments, including HCI, workflow, human oversight, test plan design, and revalidation.

Responsibilities

• Provide practice leadership for validation pathways, applied evaluation, and consultation.
• Translate approved enterprise requirements into practical validation and evaluation methods.
• Lead the development and maintenance of guidance for human-computer interaction (HCI), usability, clinical workflow, human oversight, pilot HCI design, legacy-product HCI remediation, and revalidation.
• Review Validation, Performance, and Safety content for methodological adequacy, evidence sufficiency, consistency, limitations, and alignment with approved policy and methods.
• Lead complex assessment review and consultation, calibrate AVM reviewer comments, and assure the quality and consistency of review conclusions and revalidation guidance.
• Coach AVM Engineers, Associates, and others in the department.
• Recommend required corrections, alternate methods, additional evidence, limitations, fallback or remediation strategies, and escalation.
• Translate approved enterprise validation and evaluation requirements into practical pathways and applied methods for various use cases.
• Review intended use, pathway selection, performance and safety expectations, standard-of-practice comparisons, acceptance thresholds, evaluation criteria, evidence sufficiency, limitations, and residual uncertainty.
• Evaluate the adequacy of retrospective studies, prospective designs, pilot protocols, workflow simulations, user acceptance testing, human-factors work, and alternative validation strategies.
• Review test plans/protocols, methods, and validation datasets for representativeness, traceability, production parity, functionality, robustness, calibration, subgroup and equity evidence, uncertainty, and methodological limitations.
• Review HCI, clinical workflow, usability, human oversight, automation-bias risk, patient-facing behavior, training, accessibility, guardrails, task boundaries, safe refusal, escalation, fallback, and remediation strategies.
• Lead complex case consultation and resolve non-precedent-setting method questions within approved standards, while escalating novel, precedent-setting, disputed, or out-of-method questions.
• Document traceable review conclusions, required corrections, clarification questions, alternate approaches, evidence gaps, limitations, revalidation implications, consultation needs, and escalation triggers.
• Develop and maintain validation-pathway playbooks, decision aids, reviewer rubrics, evidence examples, standard findings, consultation methods, case-library content, and escalation criteria.
• Lead reviewer training, calibration sessions, office hours, case review, quality assurance, and coaching.
• Guide remediation and revalidation approaches.
• Coordinate with clinical product teams and relevant enterprise partners.
• Convert recurring review gaps and quality findings into improved guidance, templates, evidence examples, standard assessment language, training, technology requirements, and Governance Operations enablement.
• Provide mentorship, guidance, and technical leadership to junior engineers within the AIA team.

Benefits

• Medical: Multiple plan options.
• Dental: Delta Dental or reimbursement account for flexible coverage.
• Vision: Affordable plan with national network.
• Pre-Tax Savings: HSA and FSAs for eligible expenses.
• Retirement: Competitive retirement package to secure your future.
• Continuing education and advancement opportunities.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior