Senior AI QA Engineer

Truist Bank • Charlotte, NC

1d • Onsite

About The Position

The Senior AI QA Engineer owns the quality validation of AI-enabled applications, agent workflows, and non-deterministic outputs in production-style enterprise environments. This role combines the rigor of senior software quality engineering with the specialized evaluation discipline required for AI systems where correctness, completeness, hallucination risk, bias, safety, and reliability must all be measured explicitly. This is a hands-on role focused on testing AI-powered products, agent-integrated workflows, prompt-driven solutions, and enterprise automations. The engineer designs and executes quality strategies spanning functional testing, regression testing, API validation, UI verification, output scoring, hallucination detection, evaluation dataset design, drift monitoring, and deployment gate readiness for agentic systems. The role operates within the Forge model: production-first, governance-aware, and evidence-driven. Daily work includes defining test approaches for AI workflows, building automated test assets, validating prompts and outputs, running scenario-based and adversarial checks, partnering with engineering to remediate defects, and ensuring AI-enabled capabilities meet quality, security, and operational standards before release. For this opportunity, Truist will not sponsor an applicant for work visa status or employment authorization, nor will we offer any immigration-related support for this position (including, but not limited to H-1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN-1 or TN-2, E-3, O-1, or future sponsorship for U.S. lawful permanent residence status.)

Requirements

• Systems thinker who sees the big picture and the interconnection amongst its parts.
• Ten year’s experience in corporate environment leading key initiatives
• Bachelor’s degree or equivalate experience
• Broad functional knowledge in applied enterprise information security technologies and identity management
• Excellent Communication
• Previous experience in leading complex IT projects
• Previous experience in leading infrastructure and wide-scale rollouts.
• 5+ years of QA or software test engineering experience, including experience with enterprise application testing and production release quality.
• Demonstrated experience testing AI-enabled applications, machine learning workflows, LLM-powered features, or non-deterministic output systems.
• Strong understanding of software quality disciplines including functional testing, regression testing, API testing, defect lifecycle management, and release validation.
• Experience designing or applying scoring frameworks, expected-output datasets, or benchmark-based evaluation methods for AI quality validation.
• Hands-on experience with automated testing tools and CI/CD-aligned test execution for enterprise software delivery.
• Ability to assess AI outputs for hallucination, inconsistency, incompleteness, unsafe content, or task failure using structured and repeatable methods.
• Strong communication and documentation skills with the ability to translate ambiguous AI quality failures into actionable findings for engineers and product teams.
• Experience working within enterprise governance, security, and deployment controls where evidence-based release readiness is .

Nice To Haves

• Experience testing agentic systems, multi-step orchestration workflows, or tool-calling solutions that combine prompts, APIs, and business logic.
• Experience with AI evaluation frameworks, prompt testing approaches, benchmark sets, or enterprise agent quality tooling.
• Experience with Azure, Microsoft Copilot / Copilot Studio, enterprise workflow automation, or AI-enabled productivity platforms.
• Experience with API, UI, and workflow testing across cloud-native or event-driven enterprise systems.
• Experience in financial services, cybersecurity, risk, compliance-heavy environments, or other regulated industries.
• Familiarity with observability metrics, telemetry, model change validation, and post-release quality monitoring for AI-enabled capabilities.
• Experience collaborating with AI security or adversarial testing teams on release quality, safety, and control validation.

Responsibilities

• Design and execute quality strategies for AI-powered applications, agents, prompts, and workflow automations, covering functional behavior, output quality, regression risk, and production readiness.
• Validate AI outputs for accuracy, completeness, consistency, policy adherence, and business usefulness using defined scoring rubrics, golden datasets, and repeatable evaluation methods.
• Build and maintain manual and automated test assets for UI, API, workflow, and agent-integrated solutions, including tests for prompts, tools, memory behavior, and non-deterministic outputs.
• Conduct hallucination, drift, and quality degradation testing for agentic systems and AI-enabled workflows before and after model, prompt, or data changes.
• Partner with engineering teams to investigate failures, isolate root causes, and drive remediation for defects affecting AI output quality, system behavior, or release confidence.
• Define and enforce quality gates for AI features and deployments, including acceptance thresholds for output quality, test coverage, observability, and release readiness.
• Support evaluation of tool-calling behavior, workflow branching, human-in-the-loop steps, and failure handling across complex AI or agentic user journeys.
• Create and maintain test data, scenario sets, and benchmark suites that reflect real business conditions, edge cases, and regulated enterprise requirements.
• Participate in sprint planning, grooming, demos, and retrospectives to ensure quality considerations are built into design, acceptance criteria, and release commitments from the start.
• Document defects, findings, evaluation results, and release-readiness evidence clearly enough for engineering, product, and governance stakeholders to act on quickly.
• Collaborate with AI security, product, engineering, and operations teams to ensure deployed solutions meet Forge standards for safety, quality, traceability, and auditability.
• Continuously improve AI QA methods, automation coverage, benchmark datasets, and operational test practices as the platform and use cases scale.

Benefits

• All regular teammates (not temporary or contingent workers) working 20 hours or more per week are eligible for benefits, though eligibility for specific benefits may be determined by the division of Truist offering the position.
• Truist offers medical, dental, vision, life insurance, disability, accidental death and dismemberment, tax-preferred savings accounts, and a 401k plan to teammates.
• Teammates also receive no less than 10 days of vacation (prorated based on date of hire and by full-time or part-time status) during their first year of employment, along with 10 sick days (also prorated), and paid holidays.
• For more details on Truist’s generous benefit plans, please visit our Benefits site .
• Depending on the position and division, this job may also be eligible for Truist’s defined benefit pension plan, restricted stock units, and/or a deferred compensation plan.
• As you advance through the hiring process, you will also learn more about the specific benefits available for any non-temporary position for which you apply, based on full-time or part-time status, position, and division of work.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Number of Employees: 5,001-10,000 employees