Lead Data and AI Quality Engineer
RBC•Toronto, ON
•Onsite
About The Position
We are seeking a Lead Data and AI Quality Engineering to join our Risk and Compliance team and take ownership of quality assurance for AI and Non-Financial Risk Reporting initiatives. This role combines strategic test planning with hands-on automation and manual testing expertise to ensure the delivery of robust, high-quality solutions that meet regulatory and compliance standards.
Requirements
- Extensive Python proficiency with 5+ years of hands-on development experience.
- Advanced Python expertise including OOP principles, design patterns, async programming, and advanced data structures.
- Strong software development background with experience in designing, coding, and debugging applications.
- Hands-on experience testing AI models and machine learning systems.
- Experience designing and testing AI code agents, autonomous systems, or LLM-powered applications.
- Demonstrated expertise in Large Language Model (LLM) testing, including prompt engineering, output validation, and hallucination detection.
- Experience testing Model Context Protocol (MCP) implementations and integrations.
- Knowledge of relational and non-relational databases (e.g., Oracle, SQL Server, PostgreSQL, MongoDB).
- Solid understanding of DevOps practices and containerization.
- Proficiency with Git and GitHub Actions for CI/CD pipeline automation and workflow orchestration.
Nice To Haves
- Software development certifications or advanced programming credentials.
- Experience with data governance, data quality frameworks, and metadata management.
- Experience with AI agent frameworks and orchestration tools.
- Knowledge of Infrastructure as Code (IaC) and containerization best practices.
Responsibilities
- Ensures that products being built and processes being used adhere to all standards and requirements.
- Applies extensive, in-depth knowledge, skills, and practices to perform complex assignments.
- Test Strategy & Planning: Design and implement comprehensive test strategies aligned with risk, compliance, and regulatory requirements.
- Define test scope, acceptance criteria, and quality gates for AI and risk reporting solutions.
- Collaborate with Risk and Compliance stakeholders to understand requirements and translate them into testable specifications.
- Identify quality risks and develop mitigation plans in alignment with compliance frameworks.
- Application Development & Testing: Design and code application programs for testing purposes, including test utilities and quality assurance tools.
- Perform comprehensive testing for developed applications, including functional, integration, and system-level testing.
- Validate system performance and reliability across Risk and Compliance workflows.
- Proven expertise building production server applications (APIs, microservices, backends).
- AI Code Agent Testing: Design and execute comprehensive test strategies for AI code agents and autonomous systems.
- Test AI code agents' ability to generate, review, and validate code across risk and compliance applications.
- Perform edge case testing and failure scenario analysis for AI agent decision-making processes.
- Evaluate AI agent performance metrics, including accuracy rates, false positives, and code quality standards.
- Database & Data Modeling: Validate database schemas, data models, and data integrity for risk and compliance reporting systems.
- Design and execute SQL queries to test data consistency, accuracy, and completeness across databases.
- Test Automation: Develop and maintain scalable, automated test frameworks and test suites using extensive Python expertise.
- Leverage Python libraries (pytest, requests, django, fastapi) for comprehensive test coverage.
- Execute automated tests as part of CI/CD pipelines and regression testing cycles.
Benefits
- bonuses
- flexible benefits
- competitive compensation
- commissions
- stock where applicable
- Leaders who support your development through coaching and managing opportunities
- Ability to make a difference and lasting impact.
- Work in a dynamic, collaborative, progressive, and high-performing team
- A world-class training program in financial services
- Opportunities to do challenging work
- Opportunities to take on progressively greater accountabilities
- Access to a variety of job opportunities across business and geographies
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
