AI Solutions QA Analyst

McDermott Will & Emery • Chicago, IL

5d • Remote

About The Position

McDermott, Will & Schulte LLP is building an AI program for legal practices and operational teams and seeks an AI Solutions QA Analyst to own the quality lifecycle for AI-powered solutions. The role covers testing from unit and edge-case testing through system, performance, and scalability testing, as well as integrating automated test suites into CI/CD pipelines. The analyst partners with AI Solutions Analysts and the engineering team to ensure solutions are tested, reliable, and production-ready, using AI tools to generate, evaluate, and refine test cases.

This is a remote-first role reporting to the Director of QA. The successful candidate will help establish quality standards for legal AI solutions and promote testing as a routine part of the development workflow. Full-time employees are eligible for a discretionary bonus and a comprehensive benefits package. The target hiring range is $93,000–$142,000; actual base pay may vary based on market location and job-related factors.

Requirements

• Actively use AI tools and be able to direct leading AI platforms to generate, evaluate, and refine test cases.
• At least 4 years of formal QA experience, including writing and executing test scripts across the full testing lifecycle.
• Strong experience with manual and automated unit, integration, system, and regression testing.
• Hands-on experience integrating test suites into CI/CD pipelines.
• Demonstrated ability to design boundary and edge-case tests for complex, data-driven applications.
• Understanding of performance and scalability testing concepts.
• Solid grasp of defect tracking, test documentation, and quality metrics reporting.
• Strong analytical thinking and ability to investigate untested AI behavior.
• BS in Computer Science, Information Systems, Engineering, or equivalent experience.

Nice To Haves

• Azure DevOps experience, including pipeline configuration and automated test integration.
• Experience testing AI-powered applications, such as prompt-driven systems, agentic workflows, or LLM-integrated tools.
• Familiarity with AI-assisted test-generation workflows and code-generating platforms.
• Exposure to security testing principles in collaboration with a dedicated security team.
• Experience in legal, financial services, or other regulated industry environments.
• Experience with load-testing tools.

Responsibilities

• Own the QA lifecycle for AI-powered solutions, including unit, integration, system, performance, and scalability testing.
• Partner with Analysts to generate and evaluate test cases using AI-assisted tools, particularly for boundary conditions, edge cases, and AI output validation.
• Work with engineering to define and integrate automated test suites into the Azure DevOps CI/CD pipeline for each solution.
• Define system-level test strategies with the Director of QA, covering functional, regression, and non-functional testing requirements.
• Coordinate with the security team to translate security testing guidance into solution-specific security test cases.
• Evaluate AI solution reliability across varied inputs and ensure consistent, predictable behavior under real-world conditions.
• Maintain test documentation, defect tracking, and quality metrics to provide leadership visibility into solution readiness.
• Promote a quality culture and make testing a natural part of the development workflow.

Benefits

• Comprehensive benefits package supporting physical, emotional, mental, and financial health.
• Emphasis on work-life balance.
• Competitive compensation package.
• Opportunity to earn performance-based bonuses; full-time positions are eligible for a discretionary bonus.

What This Job Offers

• Job Type: Full-time