Senior/Staff QA Engineer – Agentic AI Platform

Ontrac Solutions•Chicago, IL

16d•$80 - $90

About The Position

We are seeking a highly skilled and experienced Senior/Staff QA Engineer to support a critical customer engagement. This role is ideal for a detail-oriented tester who thrives owning quality end-to-end — someone equally comfortable writing precise manual test cases and building automated coverage for a fast-moving agentic AI product. The successful candidate will work closely with the Delivery Lead/Solutions Architect and customer engineering stakeholders to ensure product quality from initial test planning through final release.

Requirements

5+ years of professional QA experience.
At least 2+ years at a senior or staff level.
Demonstrated experience testing production software through a full Application Development Lifecycle (ADLC).
Strong understanding of the Application Development Lifecycle — from requirements through release and support.
Solid UI/UX understanding, with the ability to evaluate usability and flag design/interaction issues, not just functional bugs.
Hands-on experience testing AI Agent use cases — including conversational flows, agent decision-making, tool/function calling, and edge-case or failure-mode behavior.
Proven experience with automated testing frameworks and building maintainable, scalable test suites.
Ability to stand up and manage a Playwright cluster to run automated end-to-end (E2E) testing against a live agent platform.
Ability to independently define clear, well-structured test cases from ambiguous or evolving product requirements.
Experience driving full E2E QA lifecycle ownership: test planning, execution, defect triage, regression, and sign-off.
Deep expertise with Playwright (including cluster management and parallel execution).
Proficiency in JavaScript / TypeScript or Python for writing automation scripts.
Familiarity with testing LLM outputs, prompt evaluation frameworks, or conversational AI testing tools.
Experience integrating automated test suites into pipelines (e.g., GitHub Actions, GitLab CI, Jenkins).
Working with Docker containerization.
Proficiency with Git, Jira, and modern test management tools.

Responsibilities

Owning quality end-to-end for a fast-moving agentic AI product.
Writing precise manual test cases.
Building automated coverage.
Working closely with the Delivery Lead/Solutions Architect and customer engineering stakeholders.
Ensuring product quality from initial test planning through final release.
Testing AI Agent use cases, including conversational flows, agent decision-making, tool/function calling, and edge-case or failure-mode behavior.
Standing up and managing a Playwright cluster to run automated end-to-end (E2E) testing against a live agent platform.
Defining clear, well-structured test cases from ambiguous or evolving product requirements.
Driving full E2E QA lifecycle ownership: test planning, execution, defect triage, regression, and sign-off.
Integrating automated test suites into pipelines (e.g., GitHub Actions, GitLab CI, Jenkins).
Working with Docker containerization.