Quality Assurance Automation Engineer - AI

OnePlan Solutions • Toronto, ON

8d • Remote

About The Position

As a QA Automation Engineer - AI at OnePlan, you will play a critical role in ensuring the quality, reliability, and performance of our AI-powered features and capabilities. This role goes beyond writing test scripts. It’s about pioneering quality assurance practices for intelligent systems, defining how we validate non-deterministic AI outputs, and shaping the future of our product through rigorous automation and deep technical collaboration. You will work closely with AI engineers, product managers, and UX designers in a dynamic, agile environment to design, build, and scale automation frameworks that keep pace with our rapidly evolving AI capabilities. This is a deeply AI-focused role: the majority of your work will center on testing, evaluating, and improving AI systems, not general software, and requires both a quality engineering mindset and a genuine curiosity about how large language models and intelligent features behave in production.

Requirements

• 2-4 years of experience in QA automation engineering, with demonstrated hands-on work building and maintaining automated test frameworks.
• Proficiency with automation tools and frameworks (e.g., Playwright, Claude, or similar) and API testing tools (e.g., Postman, REST-assured).
• Demonstrated experience testing AI/ML-integrated features or services, with a solid understanding of the unique challenges of validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content.
• Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Strong analytical and critical-thinking skills, with the ability to design test cases for complex, probabilistic AI behaviors.
• Excellent verbal and written communication skills, with the ability to clearly articulate AI quality issues and their business impact to both technical and non-technical stakeholders.

Nice To Haves

• Hands-on experience with LLM evaluation techniques, including prompt testing, output scoring, and benchmarking AI model responses against defined quality criteria.
• Familiarity with Microsoft Azure AI services, Azure OpenAI, and cloud-based AI infrastructure (e.g., Azure Machine Learning, Cognitive Services).
• Experience with data-driven testing approaches, including generating and managing synthetic test datasets for training and validating AI model behavior at scale.
• Understanding of responsible AI principles, including fairness, transparency, and explainability, and experience incorporating these into QA testing practices.
• Proficiency JavaScript for scripting test utilities, data processing pipelines, and custom AI evaluation harnesses.
• Background in integrating automated QA into CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) with experience in shift-left testing and continuous quality monitoring for AI-driven products.

Responsibilities

• Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows.
• Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios.
• Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites.
• Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle.
• Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows.
• Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists.
• Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks.
• Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems.

Benefits

• Comprehensive health, dental, and vision benefits, with additional insurance options.
• Employer RRSP and 401K matching programs.
• A fun, collaborative, and diverse environment with regular health and team challenges to keep things light and enjoyable!

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level