AI Engineer

Vytalize Health•Kansas City, KS

About The Position

As an AI Engineer at Vytalize Health, you will design, build, and maintain agentic systems and LLM-powered applications that automate complex healthcare workflows and accelerate our ability to deliver data-driven clinical solutions. Working at the intersection of applied AI and healthcare, you will build agents that orchestrate data retrieval, model inference, clinical logic, and tool use to solve problems that traditionally require manual effort or specialized expertise. You will work cross-functionally with data engineering, platform, product, and clinical teams to identify high-impact opportunities for AI automation—from data source onboarding to clinical decision support to evidence synthesis. Your focus will be on building production-grade agentic systems with rigorous validation, clear confidence scoring, and human-in-the-loop oversight to ensure reliability in a regulated healthcare environment. You will establish patterns, best practices, and tooling that allow the organization to scale AI-driven automation across multiple domains. You will measure agent performance and impact—tracking accuracy, hallucination rates, and real-world clinical outcomes. You will be part of a growing AI team that values both cutting-edge AI capabilities and deep healthcare domain understanding.

Requirements

3+ years of professional experience in data engineering, backend engineering, machine learning, or a related field
1+ years of hands-on experience building with LLM APIs and agentic orchestration frameworks — not just using AI coding assistants, but architecting agentic systems
Strong Python and SQL proficiency
Experience with cloud data platforms (AWS, Databricks)
Solid understanding of data modeling, ETL/ELT patterns, and medallion architecture (Bronze/Silver/Gold)
Experience building and consuming APIs
Demonstrated experience with prompt engineering, agent evaluation, and validating LLM outputs
Experience designing evaluation frameworks, test cases, and quality assurance for AI/ML systems
Demonstrated ability to measure and track AI system performance through metrics and KPIs (accuracy, precision, recall, hallucination rates)
Strong debugging and analytical skills, especially in ambiguous or novel technical territory
Excellent written and verbal communication skills — this role requires documenting agent reasoning, decisions, and limitations clearly for both technical and non-technical audiences
Comfortable working in a fast-moving environment with incomplete information and rapidly evolving AI/ML capabilities

Nice To Haves

Experience with dbt or similar data transformation frameworks
Familiarity with orchestration tools (Airflow, Databricks Workflows) and workflow automation
Experience with agent evaluation and observability tooling (LangSmith, Langfuse, or custom frameworks)
Background in healthcare, fintech, or another regulated/high-stakes domain where AI reliability is critical
Experience building internal developer tooling, platform capabilities, or developer-facing products
Hands-on experience with RAG (retrieval-augmented generation) or other grounding techniques for LLMs
Familiarity with healthcare data formats and standards (FHIR, HL7, claims data, clinical NLP)
Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems
Understanding of healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices
Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems
Startup or high-growth environment experience with rapid iteration and learning
Published research, open-source contributions, or demonstrated thought leadership in AI/agentic systems

Responsibilities

Design, build, and maintain agentic systems and LLM-powered applications that automate healthcare workflows, data pipelines, and clinical decision support — from conception through production deployment
Build and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agentic frameworks (LangChain, LangGraph, CrewAI, or custom orchestration) to solve complex, multi-step healthcare problems
Develop prompt libraries, agent instructions, and reusable "skills" that improve agent accuracy, consistency, and reliability across different use cases and data domains
Build validation and confidence-scoring layers that flag low-confidence agent decisions for human review before production deployment; establish guardrails and review workflows for agent-authored code and outputs
Own end-to-end delivery of AI-automated systems — from problem scoping and requirements gathering through agent development, testing, and validated production deployment
Implement rigorous evaluation and QA frameworks for agentic systems — including golden datasets, test cases, output validation, hallucination detection, and regression testing
Establish and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness; measure agent accuracy, hallucination rates, clinical validity, and real-world impact
Implement observability, evaluation, and regression testing frameworks specific to agentic systems — decision tracing, lineage logging, and performance tracking
Collaborate with data engineering and platform teams to integrate agent-built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows
Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices — including explainability, auditability, and clinician trust
Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare-specific requirements (accuracy, cost, latency, regulatory alignment)
Partner with data engineering to establish robust data validation and input validation layers for agents — agents are only as good as the data they operate on
Lead experimentation and measurement of AI-automated systems impact on speed, quality, compliance, and cost across healthcare workflows
Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non-technical stakeholders
Mentor AI Connector Engineers and other team members on agentic development patterns, LLM-powered application design, and responsible AI practices
Work on-call as needed to support production agentic systems, troubleshoot agent issues, and respond to performance degradation or hallucination detection