Quality Assurance Engineer

Ochsner Health•New Orleans, LA

3d•Hybrid

About The Position

Owns the quality engineering function for an agentic conversational AI platform, safeguarding the customer experience across voice and chat interactions. The position combines hands-on assurance that releases are effective and defect-free with building, scaling, and maintaining evaluation infrastructure for continuous, measurable quality improvement. The role plans and directs quality standards for conversational AI applications and supporting systems and owns the automated evaluation platform, including synthetic and adversarial agent-to-agent conversation testing, DeepEval test suites, proprietary LLM-as-judge evaluators, and deterministic rule-based checks. It designs test plans and evaluation rubrics that translate ambiguous notions of quality into clear metrics, and correlates those metrics with real-world performance and user experience. The work is weighted approximately 70% toward quality infrastructure and evaluation engineering and 30% toward manual and exploratory QA. It includes advancing a closed-loop quality process in which evaluation findings generate test cases, refine judge rubrics, identify model and prompt improvements, and re-validate fixes. This is a hybrid position, with onsite presence expected approximately once per month; additional onsite travel may be required based on business needs.

Requirements

At least 3 years of experience in software, quality assurance and testing methodologies, and defect-tracking tools.
Conversational AI testing experience, including intent accuracy, fallback handling, and escalation logic.
Understanding of the software development lifecycle and API/integration testing.
Ability to design and execute detailed test cases across multiple scenarios.
Strong attention to detail and quality standards.
Ability to collaborate and communicate effectively with engineers and product teams to resolve defects.
Ability to document and report test outcomes.
Ability to balance testing speed with thoroughness and accuracy while maintaining consistency and discipline in the testing process.

Nice To Haves

Familiarity with AI/machine learning and natural language processing concepts, particularly for conversational AI or chatbot testing.

Responsibilities

Develop and implement quality standards, test plans, and methods for inspection, testing, and evaluation across the conversational AI platform and supporting systems.
Define functional and non-functional test strategies, including conversation flow accuracy, response quality, latency, and fallback and escalation behavior.
Design and execute test cases covering user scenarios, edge cases, and failure points, including regression, negative, usability, and integration testing.
Build and maintain automated test scripts and frameworks, and perform manual testing where appropriate.
Identify, document, and track defects through resolution, and confirm fixes.
Test the product both as intended and as misused to anticipate incorrect usage and reduce operational and compliance risk.
Maintain test documentation and report on quality assurance activities, defect patterns, and release readiness.
Own automated evaluation infrastructure, including synthetic and adversarial agent-to-agent conversation testing, DeepEval test suites, proprietary LLM-as-judge evaluators, and deterministic rule-based checks.
Design evaluation rubrics and correlate quality metrics with real-world performance and user experience.
Advance the quality platform toward a closed-loop process that generates test cases, refines rubrics, surfaces model and prompt improvement candidates, and re-validates fixes.
Perform other duties as assigned.
Comply with all policies and standards.
Remain knowledgeable about applicable laws, accreditation standards, and regulatory requirements, and ensure compliance.
Immediately report known or suspected unethical or questionable conduct and patient/employee safety, privacy, or other compliance concerns.