V&V Engineer - AI-Driven Testing & Validation

Capgemini • Dallas, TX

3d • Onsite

About The Position

At Capgemini Engineering, the world leader in engineering services, we bring together a global team of engineers, scientists, and architects to help the world’s most innovative companies unleash their potential. From autonomous cars to life-saving robots, our digital and software technology experts think outside the box as they provide unique R&D and engineering services across all industries. Join us for a career full of opportunities. Where you can make a difference. Where no two days are the same.

We are looking for a senior quality engineering leader to drive validation and testing of enterprise AI and Generative AI solutions, including LLM‑powered applications and agentic workflows. This role will define and implement modern AI testing strategies across automation, model evaluation, data quality, and responsible AI, while partnering closely with AI engineering and product teams.

Requirements

• 10+ years of professional experience in Quality Engineering and Test Automation, validating complex enterprise applications.
• Proficient in validating AI/ML systems, including Generative AI and LLM-based applications with strong proficiency in Python and experience building automation frameworks.
• Working knowledge of AI evaluation metrics, including BLEU, ROUGE, embedding‑based similarity, precision, recall, F1‑score, and human‑evaluation methodologies, with practical experience in prompt validation, agentic workflow testing, and end‑to‑end AI model evaluation.
• Experience with AI/ML frameworks and ecosystems: TensorFlow, PyTorch, LangChain, LangGraph, and LlamaIndex.
• Experience integrating automated testing into CI/CD pipelines (e.g., GitHub Actions, Jenkins, GitLab CI, Azure DevOps).

Nice To Haves

• Familiarity with AI safety, bias detection, and fairness evaluation.
• Experience in vector databases, RAG and embedding pipelines.
• Experience with MLOps and LLM observability tools (e.g., MLflow, Weights & Biases, LangSmith, Ragas, DeepEval, TruLens).
• Experience with cloud AI services on AWS/Azure/GCP.
• Experience with AI governance frameworks and regulatory standards.
• Bachelor’s or Master’s degree in Computer Science, Data Science, or a related technical field.

Responsibilities

• Lead end‑to‑end quality engineering for enterprise AI and Generative AI systems, including LLM‑powered applications, RAG pipelines, and agentic workflows.
• Design and execute prompt validation strategies, evaluating LLM outputs for accuracy, semantic relevance, hallucination risk, and safety compliance.
• Build automated AI model evaluation pipelines using metrics such as BLEU, ROUGE, embedding‑based similarity, precision, recall, and F1‑score.
• Validate agentic systems (tool usage, multi‑step reasoning, planner‑executor workflows) for correctness, determinism, and failure‑mode handling.
• Architect and maintain Python‑based automation frameworks for AI/ML model evaluation, regression testing, and continuous quality monitoring.
• Integrate AI testing into CI/CD pipelines to automatically evaluate model updates, prompt changes, and dataset revisions prior to release.
• Develop reusable test harnesses for prompt regression testing, golden‑set evaluation, A/B model comparison, and human‑in‑the‑loop review workflows.
• Perform data quality validation across training and inference pipelines using EDA, schema validation, and cross‑validation, while conducting bias and fairness analysis across demographic and contextual dimensions to ensure responsible AI outcomes.
• Drive model robustness testing through adversarial inputs, distribution‑shift detection, and edge‑case stress testing, and establish regression testing standards for retraining and fine‑tuning cycles to prevent model quality drift.
• Partner with AI engineers to validate solutions built on TensorFlow, PyTorch, LangChain, LangGraph, and LlamaIndex, while championing responsible AI principles including safety, transparency, explainability, and compliance with evolving AI governance standards

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Senior