AI Software Test Engineer

Smart IMS•Ann Arbor, MI

3d•Onsite

About The Position

As an AI Software Test Engineer, you will be responsible for designing, automating, and executing testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines software quality engineering, test automation, AI validation, and performance testing to ensure AI solutions are reliable, secure, accurate, scalable, and production-ready.

Requirements

Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field.
3–6 years of experience in software testing, QA automation, or quality engineering.
Strong programming experience in Python, Java, JavaScript/TypeScript, or C#.
Experience building automated test frameworks and API testing solutions.
Strong understanding of SDLC, Agile methodologies, test automation, and CI/CD pipelines.
Experience testing distributed systems, web applications, enterprise APIs, and cloud-based solutions.
Knowledge of AI testing concepts including prompt testing, hallucination detection, response evaluation, AI safety testing, and regression validation.
Experience with tools such as Selenium, Playwright, Postman, JUnit, PyTest, Jenkins, GitHub Actions, Docker, and Kubernetes.
Familiarity with Generative AI, LLMs, AI agents, RAG architectures, MCP integrations, and platforms such as OpenAI, Azure OpenAI, Claude, or Gemini.
Strong analytical, troubleshooting, communication, and problem-solving skills along with knowledge of responsible AI, AI governance, and AI risk management practices.

Responsibilities

Design and execute testing strategies for AI applications, machine learning models, and generative AI solutions.
Validate AI outputs for accuracy, reliability, consistency, safety, and governance compliance.
Develop and maintain automated test frameworks and evaluation pipelines for AI systems.
Create test cases for AI agents, prompt-driven workflows, RAG solutions, and intelligent automation processes.
Perform functional, integration, regression, end-to-end, and performance testing.
Conduct adversarial, negative, and edge-case testing to identify model failures and hallucinations.
Test AI APIs, integrations, MCP-based solutions, and tool-calling capabilities.
Integrate AI testing activities into CI/CD pipelines and automated delivery processes.
Execute load, stress, scalability, and resiliency testing for AI platforms and services.
Collaborate with engineers, architects, security teams, and product owners to improve software quality and production readiness.