Executive Director, AI and Digital Health Assurance

ECRI • Whitemarsh Township, PA

20h • $179,994 - $224,992 • Remote

About The Position

ECRI is building the healthcare industry's independent assurance capability for artificial intelligence. The Executive Director, AI and Digital Health Assurance designs, launches, and leads this new client-facing offering: a closed-loop practice that independently tests and validates AI-enabled solutions before deployment, surveils real-world AI safety events, investigates suspected AI-related harm, and monitors deployed systems for drift, degradation, and inequitable performance. Scope spans AI-enabled clinical decision support and large language model applications, Software as a Medical Device (SaMD) and Software in a Medical Device (SiMD), connected and wearable devices, and EHR-integrated software. The Executive Director sets ECRI's AI evaluation strategy and methodology, builds the team and technical infrastructure to execute it at scale, and carries commercial accountability for the design, launch, and growth of the offering. The role operates from a position no other organization holds: ECRI's federally listed Patient Safety Organization and its patient safety event corpus, its device hazard and alerts apparatus, and more than five decades of independent, conflict-free technology evaluation. Preserving that independence is a core obligation of the role, not an administrative formality. The underlying technology moves in months, not years. This role requires continuous, hands-on command of the frontier of AI capability, tooling, adoption patterns, and threats, sustained as an ongoing discipline rather than a body of knowledge fixed at hire.

Requirements

• 10+ years of progressive experience in healthcare technology evaluation, clinical software quality and validation, medical device or health IT safety, or a closely related discipline.
• 3+ years of hands-on leadership or delivery experience specifically involving AI- or ML-enabled systems, including meaningful direct exposure to generative AI, agentic systems, or large language model applications.
• 5+ years leading teams, with demonstrated success building a function, practice, or program from an early or undefined state rather than inheriting a mature one.
• Demonstrated experience shipping an AI-enabled product or feature into production and operating it post-launch, with direct accountability for real-world performance, user adoption, and iteration. Academic research, pilots, or third-party evaluation alone do not substitute for this.
• Deep and current command of AI and machine learning as applied in healthcare, including the distinct evaluation challenges posed by generative, agentic, and large language model systems, sustained through active use of current tools rather than fluency anchored to a single past project.
• Demonstrated experience designing and executing validation or evaluation strategies for clinical software in regulated or highly governed environments, including design controls, traceability, change control, and risk management.
• Experience building or operating technical infrastructure to exercise and test AI systems and their outputs in a controlled sandbox environment, including construction of benchmark and adversarial test sets.
• Proven client-facing experience translating provider, payer, and vendor needs into measurable evaluation plans, and communicating technical and clinical findings, including limitations, uncertainty, and risk implications, in a practical and action-oriented manner.
• Demonstrated commercial contribution: shaping offerings, scoping and pricing engagements, supporting pursuits, and growing client relationships.
• Established credibility with senior healthcare and industry leaders, evidenced by publication, standards or consensus body participation, or speaking at recognized venues.
• Advanced, current, hands-on fluency across multiple leading foundation model families (proprietary and open-weight), agent and orchestration frameworks, and model/prompt evaluation and observability tooling, maintained through active use rather than point-in-time familiarity.
• Working knowledge of AI-specific threat and risk frameworks, such as the OWASP Top 10 for LLM Applications and MITRE ATLAS.
• Working familiarity with data analysis environments such as Python, R, or SQL, sufficient to direct technical work and independently interrogate results.
• Proficient with Microsoft 365 (Word, Excel, PowerPoint) and standard collaboration tools.

Nice To Haves

• Working knowledge of patient safety event analysis and root cause methodology, or demonstrated capability leading investigations of clinical harm events.
• Experience within a Patient Safety Organization, risk management, or quality and safety function is a strong advantage.
• Experience in one or more adjacent domains, such as connected and wearable device validation, EHR-integrated clinical decision support and workflow integration, or familiarity with the health AI assurance frameworks now in market (Joint Commission's Responsible Use of AI in Healthcare certification, CHAI governance playbooks).
• ISO/IEC 42001 lead implementer or auditor, or an equivalent AI management system credential.
• RAPS RAC (US or Global) and/or ASQ CQA or CQE, or an equivalent quality systems credential.
• Certified Professional in Patient Safety (CPPS).

Responsibilities

• Design and launch the offering. Define the service architecture, methodology, delivery model, and commercial structure for ECRI's AI assurance offering, and take it from concept to an operating, revenue-generating practice spanning advisory, independent evaluation, event investigation, and ongoing monitoring.
• Carry commercial accountability. Partner with sales, marketing, and product strategy to build pipeline, serve as the senior technical authority in client pursuits, and own bookings, revenue, margin, and retention for the practice.
• Productize what repeats. Convert recurring elements of the practice into scalable products and tools, including assessment frameworks, benchmark and reference datasets, monitoring instrumentation, and certification-readiness toolkits, in partnership with product strategy and the data and analytics products team.
• Set evaluation strategy and methodology. Establish ECRI's approach to evaluating AI-enabled solutions: evidence requirements, test design, reference and adversarial datasets, data provenance and fitness-for-context, subgroup and fairness performance, acceptance criteria, and reporting standards. Require disparities to be quantified and mitigated, not asserted away. Ensure methods are defensible, reproducible, and published where doing so advances the field.
• Maintain frontier fluency and track adoption. Personally track foundation model releases, agentic architectures, and clinical AI platforms as they ship, distinguishing genuine capability advances from marketing claims. Maintain a current view of how healthcare organizations are actually deploying AI, including the shift of agentic AI from pilot to production and the prevalence of unsanctioned "shadow AI" use. Translate both into updated evaluation criteria and where ECRI's evaluation and monitoring capacity is applied.
• Build and operate the test environment. Direct the design and operation of a secure sandbox for exercising AI systems against curated, edge-case, and adversarial inputs, covering accuracy, robustness, and failure modes, and, for generative and LLM-based tools, non-determinism, confabulation, and prompt sensitivity.
• Stand up AI safety event surveillance. Working with ECRI's Patient Safety Organization and hazard reporting programs, build the intake, taxonomy, and analytic capability to collect and classify AI-related safety events, near misses, and performance failures at scale, and to generate actionable signal for clients, vendors, and the field.
• Lead investigations of suspected AI-related harm. Direct root cause analyses of events in which an AI-enabled tool is implicated, applying systems-based and just culture methods to distinguish model failure, data failure, integration failure, workflow failure, and human-automation interaction failure, and translate findings into corrective action.
• Design post-deployment monitoring and close the loop. Specify how clients detect model drift, data distribution shift, degradation following vendor updates, and erosion of clinical benefit over time, including thresholds, cadence, escalation paths, and triggers for revalidation or decommissioning. Ensure findings from surveillance and investigation systematically update ECRI's evaluation criteria, hazard communications, and published guidance, so real-world field experience continuously strengthens pre-deployment testing.
• Advise senior leaders and track the regulatory landscape. Serve as ECRI's senior authority on AI assurance to boards, executive teams, AI governance committees, and vendor executives, translating technical findings, including limitations, uncertainty, and the recommendation not to deploy, into decisions those audiences can act on. Track and interpret FDA expectations for AI-enabled device software functions, Joint Commission's Responsible Use of AI in Healthcare certification standards, CHAI governance playbooks, NIST AI RMF, and ISO/IEC 42001, ISO 14971, and IEC 62304, and position ECRI's offering to help clients meet them.
• Direct security and threat evaluation of AI-enabled systems. Oversee assessment of AI-specific threats, including prompt injection and adversarial manipulation, agentic and tool-use attack surfaces (e.g., Model Context Protocol-style integrations), model and data supply chain exposure, unsanctioned "shadow AI" use, and access control for connected and EHR-integrated deployments. Reference frameworks such as the OWASP Top 10 for LLM Applications and MITRE ATLAS, in coordination with ECRI's cybersecurity capability.
• Build and lead the team. Recruit, develop, and lead a multidisciplinary team spanning clinical, data science, engineering, human factors, and cybersecurity expertise. Establish standard operating procedures, training, quality control, and peer review so that output is consistent, reproducible, and defensible at scale.
• Protect ECRI's independence. Own and enforce the conflict-of-interest and firewall protocols that keep ECRI's evaluations free of vendor influence, including clear separation between advisory work, independent evaluation, and any vendor-facing revenue.
• Represent ECRI in external standards, consensus, and policy forums on health AI assurance, including regulatory dockets, professional societies, and industry coalitions.
• Contribute AI-related content to ECRI's flagship publications and thought leadership, including the annual Top 10 Health Technology Hazards and Top 10 Patient Safety Concerns.
• Advise ECRI's internal AI program on the safe, appropriate, and defensible use of AI within ECRI's own products, services, and operations.
• Other duties, as assigned.

Benefits

• Comprehensive healthcare coverage: Medical, dental, vision, life insurance, disability coverage, critical illness insurance, and more.
• Retirement savings: Employer-matching Retirement Savings Plan.
• Culture of Belonging: Participate in employee groups to learn from, growth with, and support colleagues in our collective success.
• Purpose beyond the paycheck: 16 paid hours annually through ECRI Cares to volunteer with preapproved charities.

What This Job Offers

• Job Type: Full-time
• Career Level: Executive