AI QA Test Lead

Truliant Federal Credit Union•Winston-Salem, NC

1d•Remote

About The Position

The AI QA Test Lead is a critical quality and risk-management leader responsible for ensuring the reliability, accuracy, and production readiness of the credit union’s AI-driven member and staff experiences. This role owns the overall AI quality assurance strategy, directly safeguarding millions of AI-powered interactions each year by ensuring solutions meet defined standards for accuracy, compliance, brand alignment, and ethical use. Serving as the primary authority on AI testing and validation, the AI QA Test Lead designs and manages comprehensive testing plans across conversational AI, generative AI, and agentic AI workflows. The role partners closely with the AI Program team and Truliant’s business stakeholders, and key vendors to validate new use cases, manage AI-related risk, and ensure all AI releases are thoroughly tested before deployment. The role manages the end-to-end correction loop with third-party vendors integral to the delivery of Truliant’s AI use cases. Key responsibilities include defining QA plans, leading end-to-end UAT, validating ethical guardrails, and establishing scalable QA practices that enable safe and confident expansion of Truliant’s AI capabilities.

Requirements

Must have a strong understanding of conversational AI, generative AI, NLP, and intent-based bot frameworks.
Must have the ability to evaluate AI responses for tone, brand voice, compliance, and member experience impact.
Must have the ability to understand credit union business processes and assess AI solutions against operational and member needs.
Must have knowledge of ethical AI principles, including bias detection, guardrail validation, and hallucination mitigation.
Must have excellent collaboration and communication skills to work across technical teams, business stakeholders, and frontline staff.
Must have the ability to translate business and regulatory requirements into clear AI testing criteria and validation scenarios.
Must have familiarity with analytics, dashboards, automated QA tools, or conversation intelligence platforms used to support testing and monitoring.
Must have strong problem-solving skills with the ability to identify systemic quality issues and propose scalable solutions.
Must have attention to detail and high standards when reviewing AI outputs, workflows, and system changes.
Must have an understanding of UX and conversational design principles as they relate to testing and validation.
Must have the ability to manage multiple priorities in a fast-paced, innovation-driven environment.
Must have strong organizational skills and the ability to work independently with minimal supervision.
Must be able to work in a team environment with strong collaborative skills.
Must have superior interpersonal skills; ability to get along with diverse personalities; tactful, mature, flexible, respectful.
Must be flexible and able to shift resources and priorities as required.
Should possess a strong commitment to providing excellent service to Truliant’s members.
Bachelor’s degree required in a relevant field such as Information Systems, Computer Science, Business Administration, or a related discipline required.
Equivalent combination of education and directly applicable work experience may be considered, particularly in AI QA, software testing, or risk-focused roles.
3–7 years of experience in quality assurance, testing, or validation roles involving complex systems, with at least 2 years focused on AI, conversational AI, or automation platforms required.
Demonstrated experience leading test planning, execution, defect management, and UAT for enterprise solutions required.
Expertise in quality assurance methodologies, test planning, test execution, defect tracking, release validation, and vendor correction loops required.
Experience testing AI systems for accuracy, edge cases, failure modes, and unintended behavior required.
Strong experience leading UAT efforts and serving as the final approval authority for production releases required.

Nice To Haves

Experience testing AI or automated decision systems, including generative AI or NLP-based solutions, strongly preferred.
Experience partnering with AI engineers, solutions architects, product owners, or business leaders to ensure quality and risk alignment preferred.
Background in financial services, credit unions, banking, or other regulated environments is a plus.

Responsibilities

Owns and leads the end-to-end quality assurance strategy for the credit union’s AI self-service and internal support solutions, with a primary focus on conversational AI platforms such as Kore.ai.
Defines, documents, and maintains AI QA standards, test frameworks, methodologies, and acceptance criteria across all AI initiatives.
Designs and executes comprehensive AI test plans covering functional accuracy, conversational flows, edge cases, integrations, compliance, and regression scenarios.
Leads and governs all AI-related user acceptance testing (UAT), serving as the final quality gate for approving AI solutions prior to production release.
Validates AI-generated responses for accuracy, consistency, brand alignment, and adherence to member-first service standards.
Tests and monitors ethical AI guardrails, including bias prevention, hallucination detection, inappropriate response handling, and escalation logic.
Partners closely with the AI Solutions Architect and AI Optimization Analyst to review solution designs and ensure testability throughout the AI development lifecycle.
Establishes testing processes for prompt changes, workflow updates, model enhancements, and new AI capabilities to ensure safe iteration and deployment.
Analyzes defects, testing outcomes, and production issues to identify root causes and drive continuous improvement in AI quality.
Collaborates with business units and frontline teams to validate AI behavior against real-world use cases and operational expectations.
Tracks, documents, and reports on AI quality metrics such as defect trends, release readiness, risk severity, and post-release issues.
Stays current on AI QA best practices, regulatory considerations, and emerging testing techniques, proactively recommending improvements to strengthen AI reliability and governance.
Should possess a strong commitment to providing excellent service to Truliant’s members.
Assists with other tasks and projects as assigned.