AI test Automation

Saxon Global•Dallas, TX

18h

About The Position

The Senior AI Test Automation Engineer designs, develops, maintains, and executes automated testing and evaluation solutions for traditional software and LLM-powered applications. In a forward-deployed capacity, the role engages directly with users and delivery teams to capture real-world usage patterns and feedback, translating them into an expanding evaluation framework that validates LLM behavior against the flows users actually follow. The role partners with delivery teams, QA, development, product, AI engineering, and the QA Center of Excellence (CoE) to establish scalable automation and evaluation practices, increase test and evaluation coverage, and integrate quality controls throughout the software delivery lifecycle—from pre-deployment regression testing through production observability. Core work includes building offline LLM evaluation suites and automated UI, API, integration, and regression tests; instrumenting LangGraph workflows with LangSmith tracing; integrating tests and evaluations into CI/CD pipelines; analyzing agent behavior and failures; and using production feedback to refine datasets, evaluators, and test coverage. The role also contributes to standards, reusable components, reporting, and continuous improvement across automation and AI evaluation practices.

Requirements

At least 5 years of experience in test automation engineering, software quality assurance, or a related role.
1–2+ years of hands-on experience testing or evaluating LLM-powered applications, including building evaluation datasets, defining pass/fail criteria for nondeterministic outputs, and using LLM-as-judge or heuristic evaluators.
Hands-on experience with LangSmith for LLM application tracing, evaluation, and observability, including creating datasets, running experiments, and configuring evaluators.
Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling.
Hands-on experience building and maintaining automated test suites using Playwright; TypeScript is preferred.
Working proficiency in Python and/or TypeScript sufficient to author custom evaluators, tracing instrumentation, and test code.
Experience automating UI and API testing, and experience with API testing tools or frameworks.
Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent, including integrating evaluation runs as pipeline gates.
Working knowledge of source-code version control, including Git.
Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies.
Ability to analyze requirements and user feedback, identify test and evaluation scenarios, and create maintainable automated coverage.
Strong problem-solving, troubleshooting, communication, and collaboration skills, including comfort engaging directly with end users to gather feedback in a forward-deployed capacity.

Nice To Haves

Ability to understand and document end-to-end business processes, user journeys, and operational workflows.
Experience partnering with business stakeholders and end users to translate process requirements into test scenarios, acceptance criteria, and LLM evaluation datasets.
Ability to identify process exceptions, edge cases, dependencies, and risks that may affect application or agent behavior.
Experience validating that automated workflows and LLM-powered features produce outcomes aligned with defined business rules and user needs.
Experience using production feedback and observed user behavior to refine process coverage, test automation, and evaluation frameworks.
Experience with online evaluations, production monitoring, and quality drift detection for LLM applications.
Experience with agent trajectory evaluation, RAG evaluation (retrieval relevance, groundedness, hallucination detection), or guardrails validation.
Familiarity with prompt engineering and prompt versioning workflows, and evaluating the impact of prompt or model changes.
Understanding of statistical approaches to nondeterministic testing, such as multiple-run sampling, confidence thresholds, and summary metrics across datasets.
Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework.
Experience with contract testing tools.
Familiarity with SAFe and Agile Release Train (ART) practices.
Experience with AI-assisted engineering practices, such as using AI coding agents to accelerate test and evaluation development.
Experience with performance, accessibility, mobile, or security test automation.
Experience with test management and defect-tracking tools, such as Azure DevOps.

Responsibilities

Engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify common flows through LLM-powered features.
Translate user feedback and production traces into curated LangSmith evaluation datasets, expanding the framework as new feedback, edge cases, and failure modes emerge.
Design, build, and maintain offline regression, benchmarking, and backtesting evaluation suites to gate prompt, model, and LangGraph workflow changes before deployment.
Develop and calibrate heuristic or code-based evaluators, LLM-as-judge evaluators, and pairwise comparisons, and validate judge reliability against human review.
Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith, ensuring useful trace coverage and metadata.
Manage annotation queues and human-in-the-loop feedback workflows, route production runs to reviewers, and feed review results into datasets and evaluator calibration.
Analyze agent trajectories and multi-step LangGraph executions—including tool calls, state transitions, and retrieval steps—to identify failure points and distinguish LLM nondeterminism from product defects.
Integrate evaluation runs into CI/CD pipelines so dataset versions, experiments, and quality thresholds provide automated feedback on relevant changes.
Support adoption of production auditing and monitoring capabilities, such as online evaluations on live traffic, quality drift detection, and alerting.
Design, develop, and implement automated UI, API, integration, and regression test scripts, including deterministic end-to-end coverage of LLM-powered application surfaces.
Integrate automated tests into CI/CD pipelines for timely feedback and continuous quality validation.
Collaborate with QA, development, product, and business teams to turn requirements, acceptance criteria, and expected agent behaviors into automated test and evaluation coverage.
Analyze and triage automation failures, distinguishing framework or script issues from product defects while accounting for expected LLM nondeterminism.
Support test data management, including evaluation dataset versioning, splits, and provenance, and help identify or resolve test-environment stability issues.
Report test execution results, evaluation experiment outcomes, automation and evaluation coverage, quality trends, and risks.
Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices.
Work with the QA CoE to align practices with enterprise standards and contribute domain-specific feedback, lessons learned, and improvement opportunities.