Principal AI/ML Engineer - Validation & Evaluation Governance

Mayo Clinic•Rochester, MN

23h

About The Position

As the Principal AI/ML Engineer - Validation & Evaluation Governance within AI Validation & Monitoring (AVM), you will serve as the enterprise subject-matter authority for validation and evaluation requirements, evidence sufficiency, and complex-case consultation. You will define and interpret standards aligned with AIA Governance Policy and patient-safety risk categories; establish retrospective and prospective pathways; set expectations for performance, safety, human-computer interaction (HCI), workflow, guardrails, production parity, material change, and revalidation; and authoritatively review the Validation, Performance, Safety, HCI, workflow, and revalidation content and evidence for methodological adequacy, traceability, limitations, and decision readiness. You will provide final AVM technical direction and precedent for high-risk, novel, disputed, patient-facing, agentic, vendor-limited, retrospective-method, and other precedent-setting questions. You will direct corrections, alternate methods, added evidence, interim controls, limitations, and escalation; convene cross-functional expertise; and convert recurring gaps into reusable standards and enablement. Providing strategic and technical leadership for enterprise validation and evaluation governance and evidence standards. Defining risk-proportionate requirements for pilots, full implementations, changes, legacy products, and other pathways. Reviewing intended use, validation strategy, performance expectations, thresholds, acceptance criteria, and residual risk. Evaluating test plans and datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, subgroup and equity evidence, and uncertainty. Reviewing safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, and related mitigations. Reviewing HCI, clinical workflow, usability, automation-bias risk, patient-facing behavior, training, accessibility, and human-use evidence. Determining when retrospective evidence is sufficient and when prospective testing, simulation, UAT, human-factors work, a pilot, or another method is required. Establishing material-change and revalidation triggers, including strengthened early monitoring when retrospective evidence replaces a pilot. Issuing traceable review outcomes: concurrence, edits, revision, additional evidence, consultation, escalation, or insufficient basis. Leading consultation and setting precedent for high-risk, novel, disputed, vendor-constrained, patient-facing, autonomous, agentic, or out-of-method cases. Coordinating with clinical product teams and enterprise partners, including Research Shield, Kern Center, AVSP/CCaTS, MCP Evaluate and Deploy, FAST, Epic, DTO, IT, Architecture, data platforms, Patient Safety, Clinical Informatics, Legal, and relevant committees. Developing policies, standards, pathway guidance, playbooks, rubrics, decision aids, evidence examples, standard findings, case libraries, and escalation criteria. Leading training, calibration, office hours, case consultation, quality assurance, and coaching for AVM staff and Governance Operations Product Leads. Translating recurring gaps into improved guidance, vendor expectations, data needs, TRex workflows, evidence objects, dashboards, and technology priorities. Communicating findings, limitations, uncertainty, and recommendations clearly to clinical, technical, operational, governance, and executive audiences. Collaborating with post-deployment governance so monitoring, reporting, significant-change, and PDRS evidence can trigger retesting, workflow evaluation, or revalidation. Setting precedent without owning the complete assessment, routine case coordination, product-team gap resolution, study execution, monitoring operations, technology build/run, clinical judgment, or final approval. Provide mentorship, guidance, and technical leadership to junior engineers. May have supervisory responsibilities.

Requirements

A master’s degree in engineering, computer science, mathematics, health science, or a related field with 7 years of relevant experience, or a bachelor’s degree with 9 years of relevant experience.
Extensive (7+ years) experience applying AI and machine learning in production healthcare environments or similar highly regulated or technology focused industries, showcasing an acute understanding of healthcare technology.
Demonstrated leadership in managing complex projects, with a proven ability to navigate intricate project requirements and deliver successful outcome.
Proven success in fostering collaboration across diverse teams and effectively communicating complex technical concepts to non-technical stakeholders.
Demonstrated expertise in cloud infrastructure environment and software development tools.
Experience working with large, complex, and heterogeneous data sets, preferably in healthcare.
Strong skills in AI/ML techniques and frameworks.
Expertise with best practices in data engineering, data science, AI Engineering, and the MLOps communities.
In-depth knowledge of healthcare domain, including clinical workflows, electronic health records, medical terminologies, regulatory requirements, and industry standards.
Demonstrated leadership in administration, education, software development, and technical reporting.
Experience mentoring and training less-experienced team members, coupled with strong interpersonal, communication, and time management skills.

Nice To Haves

A Ph.D. or other doctorate is preferred.
Experience with healthcare industry informatics standards, best practices, and common data models.
Participation in national or international standards organizations or other domain-specific professional organizations, or extensive implementation experience with common data, development, and deployment standards.
Excellent communication, collaboration, and stakeholder management skills, with the ability to effectively engage with diverse stakeholders and translate complex technical concepts and results to non-technical audiences.
Demonstrated experience leading technical/quantitative teams in a regulated environment.
Familiarity with systems or quality engineering best practices, regulatory standards, and compliance frameworks, with the ability to adapt these effectively to different project scenarios.
Demonstrated experience creating risk management files and verification/validation strategies for digital health technology products within the healthcare industry.
Demonstrated expertise in user-centered design, human factors engineering, usability testing methodologies, and evaluation across AI product development.
Ability to lead expert reviews using established usability practices and methods.
Presents findings in easy-to-understand terms for the business or clinical practice.
Strong problem-solving abilities, critical thinking skills, and a passion for driving innovation and positive change in healthcare through AI technology.
Demonstrated hands-on leadership using the TRex assessment application to review AI tools deployed in Epic, ANIMATE, or comparable clinical environments, including evidence sufficiency, validation pathways, HCI, guardrails, production parity, material change, revalidation, and precedent-setting decisions.

Responsibilities

Serve as the enterprise subject-matter authority for validation and evaluation requirements, evidence sufficiency, and complex-case consultation.
Define and interpret standards aligned with AIA Governance Policy and patient-safety risk categories.
Establish retrospective and prospective pathways.
Set expectations for performance, safety, human-computer interaction (HCI), workflow, guardrails, production parity, material change, and revalidation.
Authoritatively review the Validation, Performance, Safety, HCI, workflow, and revalidation content and evidence for methodological adequacy, traceability, limitations, and decision readiness.
Provide final AVM technical direction and precedent for high-risk, novel, disputed, patient-facing, agentic, vendor-limited, retrospective-method, and other precedent-setting questions.
Direct corrections, alternate methods, added evidence, interim controls, limitations, and escalation.
Convene cross-functional expertise and convert recurring gaps into reusable standards and enablement.
Provide strategic and technical leadership for enterprise validation and evaluation governance and evidence standards.
Define risk-proportionate requirements for pilots, full implementations, changes, legacy products, and other pathways.
Review intended use, validation strategy, performance expectations, thresholds, acceptance criteria, and residual risk.
Evaluate test plans and datasets for representativeness, traceability, limitations, production parity, functionality, robustness, calibration, subgroup and equity evidence, and uncertainty.
Review safety controls, guardrails, task boundaries, safe refusal, escalation, fallback, human oversight, and related mitigations.
Review HCI, clinical workflow, usability, automation-bias risk, patient-facing behavior, training, accessibility, and human-use evidence.
Determine when retrospective evidence is sufficient and when prospective testing, simulation, UAT, human-factors work, a pilot, or another method is required.
Establish material-change and revalidation triggers, including strengthened early monitoring when retrospective evidence replaces a pilot.
Issue traceable review outcomes: concurrence, edits, revision, additional evidence, consultation, escalation, or insufficient basis.
Lead consultation and set precedent for high-risk, novel, disputed, vendor-constrained, patient-facing, autonomous, agentic, or out-of-method cases.
Coordinate with clinical product teams and enterprise partners.
Develop policies, standards, pathway guidance, playbooks, rubrics, decision aids, evidence examples, standard findings, case libraries, and escalation criteria.
Lead training, calibration, office hours, case consultation, quality assurance, and coaching for AVM staff and Governance Operations Product Leads.
Translate recurring gaps into improved guidance, vendor expectations, data needs, TRex workflows, evidence objects, dashboards, and technology priorities.
Communicate findings, limitations, uncertainty, and recommendations clearly to clinical, technical, operational, governance, and executive audiences.
Collaborate with post-deployment governance so monitoring, reporting, significant-change, and PDRS evidence can trigger retesting, workflow evaluation, or revalidation.
Provide mentorship, guidance, and technical leadership to junior engineers.
May have supervisory responsibilities.