Senior AI Tools Testing Engineer

Select Minds•Dallas, TX

2d•Hybrid

About The Position

This hands-on senior/lead engineering role focuses on building, orchestrating, monitoring, and evaluating production-grade Generative AI and Agentic AI applications. The engineer will work with Amazon Bedrock, Bedrock AgentCore, LangGraph, LangChain, RAG, and LLM evaluation and observability tools to deliver reliable, scalable, production-ready AI applications. The role emphasizes agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring, in collaboration with engineering, QA, and platform teams. The position includes designing and deploying AI agents and multi-agent workflows; developing RAG pipelines; creating automated evaluation frameworks, datasets, benchmarks, and regression tests; and implementing observability for agent executions and LLM requests. It also includes production engineering practices such as API and enterprise data integration, CI/CD, and optimization of scalability, reliability, security, performance, inference costs, and resource use. The engineer will provide technical guidance, establish reusable frameworks and best practices, participate in design discussions, code reviews, proofs of concept, and implementation, and mentor engineers. The role is based in Dallas, Texas, with hybrid work and flexible work-from-home options. The posting describes a long-term position and lists annual compensation of $120,000–$140,000.

Requirements

10+ years of experience is listed in the job details.
Experience with LangGraph, LangChain, Amazon Bedrock, and Amazon Bedrock AgentCore.
Experience with AI agents, multi-agent orchestration, tool calling, and workflow management.
Knowledge of LLMs, RAG, embeddings, vector search, prompt engineering, context management, and LLM application development.
Experience with DeepEval, LLM-as-a-Judge techniques, evaluation metrics and benchmarking, automated evaluation, and regression testing.
Ability to assess hallucination, faithfulness, relevance, and response quality.
Experience with OpenTelemetry, LangSmith, Langfuse, LLM tracing, debugging, and performance monitoring.
Experience deploying and operating production-grade GenAI applications.
Experience with API integration, backend engineering, CI/CD, and automated testing integration.
Knowledge of scalability, reliability, security, and performance optimization.
Practical experience building and evaluating AI applications; the posting prefers hands-on experience over experience limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.
Strong problem-solving, communication, and technical collaboration skills.

Nice To Haves

Hands-on experience developing and deploying GenAI or Agentic AI applications.
Experience with enterprise-scale AI implementations.
Familiarity with AWS cloud services and enterprise AI security practices.
Experience establishing AI quality standards and evaluation methodologies.

Responsibilities

Design, develop, and deploy production-grade agentic and generative AI applications.
Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
Implement multi-agent workflows, tool calling, state management, memory, and orchestration.
Create scalable LLM-powered applications and reusable AI components.
Design and implement Retrieval-Augmented Generation solutions and pipelines for ingestion, chunking, embedding, retrieval, reranking, and context management.
Optimize retrieval accuracy, response quality, latency, and cost; apply grounding, hallucination reduction, and context-relevance techniques.
Build evaluation frameworks for LLM applications and agentic workflows, including automated evaluation with DeepEval and other evaluation tools.
Use LLM-as-a-Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
Create evaluation datasets, test scenarios, benchmarks, and regression-testing pipelines.
Evaluate agent behavior, tool execution, workflow accuracy, and response consistency; identify failure patterns and improve application quality.
Implement tracing, monitoring, and observability using OpenTelemetry, LangSmith, and Langfuse.
Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance; establish debugging and monitoring practices.
Deploy AI applications using Amazon Bedrock and Bedrock AgentCore, integrating foundation models, agent frameworks, APIs, and enterprise data sources.
Apply production-readiness practices for scalability, reliability, security, and performance; support CI/CD integration and automated evaluation.
Optimize model selection, inference costs, latency, and resource utilization.
Provide technical guidance to engineering and QA teams and establish reusable frameworks, implementation patterns, and best practices.
Collaborate with architecture, DevOps, QA, and engineering teams; participate in design discussions, code reviews, proof-of-concept development, and implementation.
Mentor engineers and help teams adopt effective AI development and testing practices.