AI Solutions QA Analyst

MWE • Washington, DC

7d • $93,000 - $142,000 • Remote

About The Position

McDermott Will & Schulte LLP is building an AI program for legal practices and operational teams and seeks an AI Solutions QA Analyst to own the quality lifecycle for AI-powered solutions. The role spans unit and integration testing, system testing, performance and scalability testing, and integration of automated tests into CI/CD pipelines. The analyst partners with AI Solutions Analysts and engineers to ensure solutions are reliable and production-ready, using AI tools to generate, evaluate, and refine tests.

The position works with the Director of QA to define system-level testing strategies and quality requirements, and with engineering to integrate test suites into Azure DevOps. It also involves coordinating with the security team on solution-specific security test cases, evaluating reliability across varied inputs, and maintaining documentation, defect tracking, and quality metrics. This is a remote-first role with a technically rigorous team.

Requirements

• Actively use AI tools and be able to direct leading AI platforms to generate, evaluate, and refine test cases.
• At least 4 years of formal QA experience, including writing and executing test scripts across the full testing lifecycle.
• Strong experience with unit, integration, system, and regression testing, both manual and automated.
• Hands-on experience integrating test suites into CI/CD pipelines.
• Ability to design boundary and edge-case tests for complex, data-driven applications.
• Understanding of performance and scalability testing concepts.
• Knowledge of defect tracking, test documentation, and quality metrics reporting.
• Strong analytical thinking and ability to investigate untested AI behavior.
• Bachelor of Science in Computer Science, Information Systems, Engineering, or equivalent experience.

Nice To Haves

• Azure DevOps experience, including pipeline configuration and automated test integration.
• Experience with load-testing tools.
• Experience testing AI-powered applications, including prompt-driven systems, agentic workflows, or LLM-integrated tools.
• Familiarity with AI-assisted test-generation workflows and code-generating platforms.
• Exposure to security testing principles in collaboration with a dedicated security team.
• Experience in legal, financial services, or another regulated industry.

Responsibilities

• Own the QA lifecycle for AI-powered solutions, including unit, integration, system, performance, and scalability testing.
• Partner with Analysts to generate and evaluate test cases using AI-assisted tools, focusing on boundary conditions, edge cases, and AI output validation.
• Work with engineering to define and integrate automated test suites into the Azure DevOps CI/CD pipeline for each solution.
• Define system-level test strategies with the Director of QA covering functional, regression, and non-functional testing requirements.
• Coordinate with the dedicated security team to translate security testing guidance into solution-specific security test cases.
• Evaluate AI solution reliability across varied inputs to ensure consistent, predictable behavior under real-world conditions.
• Maintain test documentation, defect tracking, and quality metrics to provide leadership with visibility into solution readiness.
• Promote a quality culture across the delivery team and embed testing in the development workflow.

Benefits

• Competitive total rewards package.
• Opportunity to earn performance-based bonuses.
• Full-time positions are eligible for a discretionary bonus and a comprehensive benefits package.
• Benefits program supports physical, emotional, mental, and financial health, with an emphasis on work-life balance.
• Generous compensation packages.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level