Lead Architect, AI Quality and Reliability Engineering

Vizient•Edina, MN

1d•$135,200 - $236,600

About The Position

In this role, you will lead Quality Engineering and product validation for an AI-enabled healthcare operations analytics product that integrates EHR, ERP, and other health system data to deliver standardized ontologies, operational metrics, analytics, and AI-powered workflows. You will serve as a hands-on technical leader, ensuring accuracy, reliability, traceability, and usability across the product lifecycle, from data acquisition and transformation through ontology and metric computation to analytics, AI-generated insights, and end-user workflows. You will establish scalable automation, validation frameworks, and quality engineering practices for solutions built on Palantir Foundry and AIP.

Requirements

7 or more years of relevant experience required.
Strong hands-on programming and automation skills using Python, Java, TypeScript/JavaScript, or similar languages, including experience building test frameworks, validation services, or engineering tooling.
Experience validating complex data pipelines, ETL/ELT processes, transformations, analytical datasets, APIs, business rules, and calculated metrics, with strong SQL and data analysis skills.
Experience designing automated data quality, reconciliation, source-to-target validation, regression testing, and end-to-end product validation approaches.
Experience testing modern data-intensive and full-stack applications and integrating automated testing and quality gates into CI/CD pipelines.
Experience evaluating AI-enabled applications using approaches such as LLM evaluation, prompt testing, RAG or grounding validation, and agentic workflow testing.
Experience with observability, telemetry, monitoring, cloud platforms, and modern DevSecOps practices.
Strong ownership, technical leadership, problem-solving, and collaboration skills, with the ability to operate effectively in ambiguity and influence cross-functional teams.

Nice To Haves

Relevant degree preferred.
Experience with Palantir Foundry/AIP or comparable enterprise data and AI platforms preferred.

Responsibilities

Lead the end-to-end Quality Engineering and product validation strategy across data pipelines, ontologies, metrics, APIs, analytics, AI capabilities, and user-facing workflows.
Design and implement scalable automated validation frameworks for data ingestion, transformations, ontology mappings, business rules, metric calculations, APIs, application workflows, and AI-generated outputs.
Establish data quality and reconciliation capabilities that validate completeness, accuracy, consistency, source-to-target mappings, lineage, and transformation logic across EHR, ERP, and other healthcare data sources.
Develop validation strategies for ontologies and operational metrics to ensure concepts, mappings, calculations, aggregations, and analytical outputs remain accurate and consistent as the product evolves.
Lead quality engineering for AI-enabled capabilities, including LLM evaluation, prompt and workflow testing, grounding and factuality validation, regression testing, agentic workflows, and human-in-the-loop evaluation.
Build reusable test harnesses, representative datasets, automated regression suites, quality gates, and observability capabilities that enable teams to deliver solutions with confidence.
Partner with Product, Software Engineering, Data Engineering, AI Engineering, and healthcare domain experts to translate product requirements into measurable acceptance criteria and drive complex quality issues through root cause analysis and resolution.
Advance a shift-left quality culture by mentoring engineers and establishing practices for testability, automation, data quality, reliability, and AI quality while remaining hands-on with critical product validation capabilities.