AI Agentic Tester

Delan Associates • Denver, MO

5d • Remote

About The Position

We are seeking an experienced AI Agentic Tester to join our team. This role involves performing functional testing of AI agent workflows and end-to-end AI applications. You will be responsible for validating LLM responses for accuracy, relevance, consistency, and hallucination rates. The position requires testing RAG pipelines, prompt engineering, and AI agent orchestration. You will also execute API testing using Postman, Swagger, and REST APIs, and develop/maintain automation scripts using Python, Selenium, Playwright, or PyTest. A key part of this role is validating multi-agent interactions, tool calling, and workflow execution, applying AI testing methodologies and evaluation metrics to ensure response quality. Experience working with AI platforms like Azure OpenAI, AWS Bedrock, or Gemini is essential for validating AI solutions.

Requirements

• AI Agentic Testing
• Functional Testing
• Generative AI Testing
• Large Language Models (LLMs)
• AI Agents & Agentic AI
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering Validation
• AI Model Validation & Evaluation
• API Testing (Postman, REST APIs, Swagger)
• Python
• Test Automation (Selenium / Playwright / PyTest)
• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
• End-to-End Workflow Testing
• Multi-Agent Testing
• Azure OpenAI / AWS Bedrock / Gemini

Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed