Automation QA Engineer

American IT Systems • Alpharetta, GA

6d • Onsite

About The Position

We are seeking an experienced Automation QA Engineer to join our team. In this role, you will be responsible for designing and executing test cases for multi-turn agent conversations, building evaluation frameworks to assess LLM quality, and automating API and UI testing. You will also create regression suites for agentic outputs and drive test coverage and quality metrics. The ideal candidate will have experience developing automated testing frameworks using Python/JavaScript, Pytest, and Selenium, validating AI/LLM responses and agent behavior, and tracking defects using Jira or similar tools.

Requirements

• 5–7+ years of experience as an Automation QA Engineer
• Proficiency in Python/JavaScript
• Experience with Test Automation frameworks
• Experience with Pytest/Selenium
• Experience with LLM Evaluation
• Experience with Jira/Test Management tools

Responsibilities

• Design and execute test cases for multi-turn agent conversations
• Build evaluation frameworks to assess LLM quality
• Automate API and UI testing
• Create regression suites for agentic outputs
• Drive test coverage and quality metrics
• Develop automated testing frameworks using Python/JavaScript, Pytest, Selenium, or similar technologies
• Validate AI/LLM responses and agent behavior against defined quality criteria
• Track defects and test results using Jira or other test-management tools

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed