Lead Data and AI Quality Engineer

Royal Bank of Canada • Toronto, ON

10d • Onsite

About The Position

We are seeking a Lead Data and AI Quality Engineering to join our Risk and Compliance team and take ownership of quality assurance for AI and Non-Financial Risk Reporting initiatives. This role combines strategic test planning with hands-on automation and manual testing expertise to ensure the delivery of robust, high-quality solutions that meet regulatory and compliance standards.

Requirements

• Extensive Python proficiency with 5+ years of hands-on development experience
• Advanced Python expertise including OOP principles, design patterns, async programming, and advanced data structures
• Strong software development background with experience in designing, coding, and debugging applications
• Hands-on experience testing AI models and machine learning systems
• Experience designing and testing AI code agents, autonomous systems, or LLM-powered applications
• Demonstrated expertise in Large Language Model (LLM) testing, including prompt engineering, output validation, and hallucination detection
• Experience testing Model Context Protocol (MCP) implementations and integrations
• Knowledge of relational and non-relational databases (e.g., Oracle, SQL Server, PostgreSQL, MongoDB)
• Solid understanding of DevOps practices and containerization
• Proficiency with Git and GitHub Actions for CI/CD pipeline automation and workflow orchestration

Nice To Haves

• Software development certifications or advanced programming credentials
• Experience with data governance, data quality frameworks, and metadata management
• Experience with AI agent frameworks and orchestration tools
• Knowledge of Infrastructure as Code (IaC) and containerization best practices

Responsibilities

• Ensures that products being built and processes being used adhere to all standards and requirements. Applies extensive, in-depth knowledge, skills, and practices to perform complex assignments.
• Design and implement comprehensive test strategies aligned with risk, compliance, and regulatory requirements
• Define test scope, acceptance criteria, and quality gates for AI and risk reporting solutions
• Collaborate with Risk and Compliance stakeholders to understand requirements and translate them into testable specifications
• Identify quality risks and develop mitigation plans in alignment with compliance frameworks
• Design and code application programs for testing purposes, including test utilities and quality assurance tools
• Perform comprehensive testing for developed applications, including functional, integration, and system-level testing
• Validate system performance and reliability across Risk and Compliance workflows
• Proven expertise building production server applications (APIs, microservices, backends)
• Design and execute comprehensive test strategies for AI code agents and autonomous systems
• Test AI code agents' ability to generate, review, and validate code across risk and compliance applications
• Perform edge case testing and failure scenario analysis for AI agent decision-making processes
• Evaluate AI agent performance metrics, including accuracy rates, false positives, and code quality standards
• Validate database schemas, data models, and data integrity for risk and compliance reporting systems
• Design and execute SQL queries to test data consistency, accuracy, and completeness across databases
• Develop and maintain scalable, automated test frameworks and test suites using extensive Python expertise
• Leverage Python libraries (pytest, requests, django, fastapi) for comprehensive test coverage
• Execute automated tests as part of CI/CD pipelines and regression testing cycles

Benefits

• A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
• Leaders who support your development through coaching and managing opportunities
• Ability to make a difference and lasting impact.
• Work in a dynamic, collaborative, progressive, and high-performing team
• A world-class training program in financial services
• Opportunities to do challenging work
• Opportunities to take on progressively greater accountabilities
• Access to a variety of job opportunities across business and geographies

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed