Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

Alliance Health•Durham, NC

7h•$107,607 - $137,198•Hybrid

About The Position

The AI Quality Engineer ensures the reliability, accuracy, and safety of AI-driven solutions throughout development and production. The role designs evaluation frameworks for LLMs, agentic systems, and intelligent automation, helping ensure AI models and agents behave predictably, follow complex workflows, and meet standards for clinical and operational healthcare environments. This is a full-time hybrid position based in Morrisville, North Carolina. The selected candidate must reside within 50 miles of Morrisville and come to the home office on Wednesdays and for business meetings as needed.

Requirements

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university.
Three years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions.
Experience integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools in production for internal business operations and/or customer-facing applications.
At least two years of experience using Python and related machine learning and artificial intelligence libraries and frameworks to develop, test, deploy, and support AI/ML solutions.
Knowledge of AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators for agentic systems.
Ability to evaluate reasoning, tool use, and workflow reliability in agentic workflows and LLMs using proprietary and open-source models.
Experience using AI observability and tracing platforms such as LangSmith to monitor agent runs and analyze production traces.
Python programming and familiarity with libraries used for automated testing and evaluation.
Knowledge of evaluating retrieval-augmented generation (RAG) architectures.
Knowledge of LLMOps and lifecycle management, including prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs.
Ability to use AI tools such as GitHub Copilot to accelerate testing automation, documentation, and quality engineering tasks.
Ability to communicate complex AI performance metrics and evaluation findings as actionable insights for clinical, operational, and engineering stakeholders.
Must reside within 50 miles of Morrisville, North Carolina, and attend the Morrisville home office on Wednesdays and for business meetings as needed.
Employment is contingent upon satisfactory background and credit checks after offer acceptance and before the start date.

Responsibilities

Design and implement scalable AI Quality Engineering strategies for LLM solutions, agentic systems, and enterprise AI platforms.
Develop and maintain AI evaluation frameworks, including validation datasets, quality scoring methodologies, and automated testing workflows.
Run LLM-as-judge, code-based, and multi-turn evaluators on production traces.
Calibrate LLM judges so assessments align with human expert preferences and clinical accuracy.
Help determine effective system prompts and model selections for specific use cases.
Conduct side-by-side performance analyses to measure the impact of agent-component changes and prevent regressions.
Trace and visualize agent runs and conversations using distributed tracing and specialized AI assistants to identify failure modes.
Identify usage patterns and common failure modes and provide executive-level summaries of system reliability.
Partner with AI Engineering, AIOps, and Security early in development to build quality into system architecture.
Enable clinical, operational, and data subject matter experts to review agent outputs and annotate traces as part of human-in-the-loop validation.
Coordinate testing activities across agile delivery teams, including sprint planning, defect management, and release readiness.
Integrate quality controls with observability tools, monitor model drift, and assist with incident response.
Use telemetry and distributed tracing to support high availability and performance in production.