Senior AI Tools Testing Engineer

Select Minds LLC•Dallas, TX

3d•$120,000 - $140,000•Hybrid

About The Position

This is a hands-on senior/lead engineering role focused on building, orchestrating, monitoring, and evaluating production-grade Generative AI and Agentic AI applications. The position emphasizes agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring, using technologies including Amazon Bedrock, Bedrock AgentCore, LangGraph, LangChain, RAG, and LLM observability and evaluation frameworks. The engineer will collaborate with engineering, QA, platform, architecture, and DevOps teams to establish practices for building, testing, monitoring, and improving AI-powered applications. The work includes developing agentic and generative AI systems, RAG pipelines, automated evaluation and regression testing, observability, production deployment, and reusable engineering patterns. The role also provides technical guidance and mentorship to engineering and QA teams. The position is based in Dallas, Texas, with hybrid work and flexible work-from-home options.

Requirements

Hands-on experience building, orchestrating, monitoring, and evaluating production-grade Generative AI and Agentic AI applications.
Experience with LangGraph, LangChain, Amazon Bedrock, Bedrock AgentCore, AI agents, multi-agent orchestration, tool calling, and workflow management.
Knowledge of LLMs, RAG, embeddings, vector search, prompt engineering, context management, and LLM application development.
Experience with DeepEval, LLM-as-a-Judge techniques, evaluation metrics and benchmarking, automated evaluation, and regression testing.
Ability to assess hallucination, faithfulness, relevance, and response quality.
Experience with OpenTelemetry, LangSmith, Langfuse, LLM tracing, debugging, and performance monitoring.
Experience deploying and operating production-grade GenAI applications, API integration, backend engineering, CI/CD, and automated testing integration.
Knowledge of scalability, reliability, security, and performance optimization for production applications.
Strong problem-solving, communication, and technical collaboration skills.
Practical experience building and evaluating AI applications; experience limited to AI strategy, architecture documentation, conventional QA automation, or AI coding assistants is not the preferred profile.

Nice To Haves

8 or more years of software engineering or IT experience.
Experience with enterprise-scale AI implementations.
Familiarity with AWS cloud services and enterprise AI security practices.
Experience establishing AI quality standards and evaluation methodologies.

Responsibilities

Design, develop, and deploy production-grade agentic and generative AI applications.
Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
Implement multi-agent workflows, tool calling, state management, memory, and orchestration.
Create scalable LLM-powered applications and reusable AI components.
Design and implement Retrieval-Augmented Generation (RAG) solutions.
Develop document ingestion, chunking, embedding, retrieval, reranking, and context-management pipelines.
Optimize retrieval accuracy, response quality, latency, and cost; apply grounding, hallucination-reduction, and context-relevance techniques.
Build evaluation frameworks for LLM-based applications and agentic workflows, including automated evaluation with DeepEval and other evaluation tools.
Use LLM-as-a-Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
Create evaluation datasets, test scenarios, benchmarks, and regression-testing pipelines.
Evaluate agent behavior, tool execution, workflow accuracy, and response consistency; identify failure patterns and improve application quality.
Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
Establish debugging mechanisms for multi-step agentic workflows and monitoring practices to detect performance degradation and operational issues.
Deploy AI applications using Amazon Bedrock and Bedrock AgentCore, integrating foundation models, agent frameworks, APIs, and enterprise data sources.
Apply production-readiness practices for scalability, reliability, security, and performance; support CI/CD integration and automated evaluation.
Optimize model selection, inference costs, latency, and resource utilization.
Provide technical guidance to engineering and QA teams and establish reusable frameworks, implementation patterns, and best practices.
Collaborate with architecture, DevOps, QA, and engineering teams; participate in design discussions, code reviews, proof-of-concept development, and implementation.
Mentor engineers and help teams adopt effective AI development and testing practices.