Test Automation Engineer (Gen AI)

Capgemini • Dallas, TX

1d

About The Position

We are seeking a hands-on Test Automation Lead to drive quality engineering for GenAI-powered applications (LLM/agentic experiences, RAG workflows, and AI-assisted user journeys). You will design and implement automation-first validation strategies to ensure reliability, safety, and performance of AI features as requirements evolve rapidly. This individual contributor role blends deep test automation expertise with practical experience testing GenAI behaviors, integrations, and data flows, while partnering closely with product, engineering, DevOps, and AI/ML teams

Requirements

• Strong experience building and scaling automation frameworks (e.g., Selenium/Playwright/Cypress or equivalent) with maintainable design patterns and coding standards.
• Proven CI/CD integration experience (e.g., Jenkins/GitHub Actions/GitLab CI), including continuous testing and quality gates.
• Demonstrated experience testing GenAI applications (LLM features, agentic workflows, NLP-driven experiences), including evaluating response quality and workflow correctness.
• Strong programming skills in Python, Java, or JavaScript/TypeScript, with ability to build utilities, harnesses, and integrations.
• Experience with API testing/automation (e.g., Postman, RestAssured, ReadyAPI or equivalent) and integration testing across services.
• Familiarity with ALM/test management and defect tracking tools such as JIRA and related plugins (e.g., Zephyr/QMetry).
• Strong collaboration and communication skills; able to translate quality risks into actionable recommendations and align across product, engineering, DevOps, and AI/ML teams.

Responsibilities

• Automate evaluations, smoke tests, integration chains, and performance micro-tests in CI/CD.
• Build and maintain test harnesses for GenAI applications, including repeatable evaluation suites for AI outputs and end-to-end workflow validation.
• Integrate automated tests with pipelines and dashboards, providing clear quality signals (pass/fail, coverage, stability, trend analytics) to release decision-makers.
• Develop and maintain automated test suites across UI, API, and data layers, ensuring scalable and maintainable coverage.
• Partner with engineering and AI/ML stakeholders to validate prompting, orchestration, grounding, and tool integrations, and to identify failure modes early (shift-left).
• Create and maintain testing documentation (test strategy, harness design, pipeline integration, runbooks) and contribute to continuous improvement of QE standards.
• Support the program’s evolving requirements by rapidly adapting automation scope, prioritizing high-risk areas, and improving regression speed and reliability.

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed