AI/ML Engineer - Validation & Evaluation

Mayo Clinic•Rochester, MN

21h

About The Position

AI/ML Engineers at AI Validation & Monitoring (AVM) apply data, systems, computer science, clinical workflow, and governance expertise to help ensure that clinical AI products are supported by proportionate, traceable, and technically sound validation evidence. They work with AIA Governance Operations, clinicians, product teams, human-factors and user-experience specialists, data scientists, IT, architecture, patient safety, legal and regulatory partners, vendors, and other stakeholders to translate AIA Governance Policy and approved validation methods into clear evidence expectations and review conclusions. As the AI/ML Engineer - Validation & Evaluation, with the functional assignment of Validation Guidance, Guardrails, and Test Evidence, you will perform routine and moderately complex subject-matter review of the Validation, Performance, and Safety content and supporting evidence prepared. You will apply approved rubrics, evidence-sufficiency criteria, checklists, templates, and standard methods; identify missing evidence, unsupported conclusions, methodological limitations, and inconsistencies; document corrections, clarification questions, limitations, and escalation triggers; and draft traceable AVM concurrence or consultation comments. High-risk, novel, disputed, exceptional, or out-of-method questions are escalated to the Senior or Principal AI/ML Engineer comparisons, residual risk, and whether assessment conclusions are supported by the available evidence. Reviewing validation strategy and pathway selection, including retrospective analysis, prospective testing, simulation, user acceptance testing, human-factors evaluation, pilots, and other risk-proportionate methods, for alignment with intended use, workflow, and patient-safety risk. Evaluating performance expectations, metrics, thresholds, acceptance criteria, standard-of-practice Reviewing test plans and validation datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, uncertainty, bias, subgroup and equity evidence, and the adequacy of retesting evidence. Reviewing risk and safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, training, and other mitigations needed for the intended clinical or operational use. Evaluating human-computer interaction, clinical workflow, usability, automation-bias risk, patient-facing behavior, accessibility, and human-use evidence, and identifying when additional evaluation, clarification, or remediation is required. Assessing whether prior validation remains applicable after changes to the model, data, thresholds, workflow, population, interface, autonomy, guardrails, training, or intended use, and documenting needs for targeted retesting, re-evaluation, or revalidation. Producing clear, traceable review findings and recommended outcomes, including concurrence, concurrence with edits, revision required, additional evidence, consultation, escalation, or insufficient basis, and communicating technical findings to Product Leads and other technical and non-technical stakeholders. Maintaining and applying test-plan, validation-dataset, guardrail, subgroup, and production-parity templates; policy-to-evidence crosswalks; evidence examples; standard finding and clarification language; and recurring-gap themes that improve guidance, training, tools, and Product Lead self-sufficiency. This vacancy is not eligible for sponsorship/ we will not sponsor or transfer visas for this position. Also, Mayo Clinic DOES NOT participate in the F-1 STEM OPT extension program.

Requirements

A master’s degree in engineering, computer science, mathematics, health science, or a related field and 1 year experience, or a bachelor’s degree with 3 years of experience.
Experience applying AI and machine learning in production environments or similar highly regulated or technology focused industries, showcasing an understanding of healthcare technology.
Skill in cloud infrastructure environment and software development tools.
Experience working with large, complex, and heterogeneous data sets, preferably in healthcare.
Skill in AI/ML techniques and frameworks.
History of collaborating across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
Familiarity with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
Strong interpersonal, communication, and time management skills.

Nice To Haves

A Ph.D. or other doctorate degree is preferred.
Knowledge of the healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
Expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development.
Ability to conduct expert reviews using established usability practices and methods.
Presents findings in easy-to-understand terms for the business or clinical practice.
Ability to articulate complex technical concepts to diverse audiences, facilitating clear understanding and engagement from technical and non-technical stakeholders.
Ability to manage a varied workload of projects with multiple priorities and stay current on healthcare trends.
Experience with healthcare industry informatics standards, best practices, and common data models.
Demonstrated hands-on experience conducting clinical AI validation and evaluation assessments using with TRex for products deployed into clinical environments such as EPIC or ANIMATE, including test-plan review, production-parity assessment, guardrail evaluation, human-factors evidence, and revalidation decisions.

Responsibilities

Perform routine and moderately complex subject-matter review of the Validation, Performance, and Safety content and supporting evidence prepared.
Apply approved rubrics, evidence-sufficiency criteria, checklists, templates, and standard methods.
Identify missing evidence, unsupported conclusions, methodological limitations, and inconsistencies.
Document corrections, clarification questions, limitations, and escalation triggers.
Draft traceable AVM concurrence or consultation comments.
Review validation strategy and pathway selection, including retrospective analysis, prospective testing, simulation, user acceptance testing, human-factors evaluation, pilots, and other risk-proportionate methods, for alignment with intended use, workflow, and patient-safety risk.
Evaluate performance expectations, metrics, thresholds, acceptance criteria, standard-of-practice.
Review test plans and validation datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, uncertainty, bias, subgroup and equity evidence, and the adequacy of retesting evidence.
Review risk and safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, training, and other mitigations needed for the intended clinical or operational use.
Evaluate human-computer interaction, clinical workflow, usability, automation-bias risk, patient-facing behavior, accessibility, and human-use evidence, and identifying when additional evaluation, clarification, or remediation is required.
Assess whether prior validation remains applicable after changes to the model, data, thresholds, workflow, population, interface, autonomy, guardrails, training, or intended use, and documenting needs for targeted retesting, re-evaluation, or revalidation.
Produce clear, traceable review findings and recommended outcomes, including concurrence, concurrence with edits, revision required, additional evidence, consultation, escalation, or insufficient basis, and communicating technical findings to Product Leads and other technical and non-technical stakeholders.
Maintain and apply test-plan, validation-dataset, guardrail, subgroup, and production-parity templates; policy-to-evidence crosswalks; evidence examples; standard finding and clarification language; and recurring-gap themes that improve guidance, training, tools, and Product Lead self-sufficiency.