Agentic SDET I

RBC • Toronto, ON

4d • Onsite

About The Position

We’re looking for an experienced SDET with strong hands-on experience in agentic AI systems who will contribute to accelerating agentic AI adoption across RBC. This role offers the opportunity to build production-grade solutions, work collaboratively with Lines of Business teams, and develop deep expertise in the Borealis agentic AI platform. At RBC Borealis, you’ll be joining a team that bridges Lines of Business with our cutting-edge Borealis agentic AI platform. You’ll work directly with business stakeholders, platform engineers, and technical teams to implement high-impact solutions while building your capability in emerging agentic technologies with guidance from our Principal Engineers.

Requirements

• 3+ years of Quality Engineering experience in diverse environments (cloud, distributed, APIs, databases, AI/ML systems), with proven ability to resolve complex cross-functional issues in agent-based architectures.
• 3–5 years of hands-on automation experience designing efficient test strategies for autonomous systems, reducing manual testing through intelligent agents, and covering component, integration, and end-to-end scenarios.
• Deep expertise in testing AI agent applications: Proficiency in testing LLM-based agents, multi-agent systems, prompt engineering validation, output consistency testing, and handling non-deterministic behavior.
• AI framework proficiency: Hands-on experience with LangChain, LangGraph, CrewAI, AutoGen, or similar frameworks for building and testing agent workflows.
• DevOps proficiency in CI/CD, GitHub Actions, JIRA, and qTest, with experience integrating AI model evaluation into deployment pipelines.
• Automation tools expertise in Playwright, Locust, Python httpx, and AI-specific evaluation tools like promptfoo
• Programming skills in Python, TypeScript, SQL, and source control tools (Git), with strong understanding of asynchronous programming for agent orchestration.
• Experience in Agile/Iterative development, defining test strategies for AI systems, evaluating LLM outputs, and reviewing artifacts (agent code, prompt templates, automation scripts).

Nice To Haves

• Knowledge of prompt engineering techniques: Including few-shot learning, chain-of-thought prompting, ReAct patterns, and validation of prompt robustness.
• Knowledge in AI observability: Experience with platforms like Langfuse or Arize Phoenix for tracing agent execution, debugging multi-step reasoning, analyzing LLM calls, and monitoring performance metrics.
• Understanding of Retrieval Augmented Generation (RAG) pipelines and testing strategies for vector databases, embedding quality, and retrieval accuracy.

Responsibilities

• Pioneer the future of autonomous testing by designing and implementing intelligent test systems across all levels (unit, integration, end-to-end), becoming the go-to expert on AI-driven testing procedures, agent orchestration, and risk mitigation in cutting-edge applications.
• Shape groundbreaking test strategies for AI systems: Take ownership of test specifications for LLM-powered applications, architect innovative prompt validation frameworks, and establish observability practices that push the boundaries of quality assurance.
• Engineer next-generation test solutions using cutting-edge tools like Playwright, Stagehand, Locust, and Allure, seamlessly integrated with LLM orchestration frameworks to create self-healing, adaptive test systems that evolve alongside your applications.
• Build and command fleets of intelligent SDET Agents that revolutionize testing workflows—agents that generate their own test cases, dynamically validate prompts, and autonomously triage defects with unprecedented efficiency.
• Orchestrate the complete lifecycle of AI application testing: Lead autonomous test execution, harness AI-driven defect detection and classification, unlock actionable insights through LLM-powered analysis, and establish world-class observability practices.
• Drive innovation through collaboration: Partner closely with AI/ML engineers, prompt engineers, and developers to ensure bulletproof testing of intelligent systems, pushing reliability and performance to new heights in unexplored territory.
• Build the knowledge foundation for tomorrow: Document test cases for LLM behaviors and agent interaction patterns with crystal clarity, creating a knowledge base that empowers your team and advances the state of the art in AI testing.

Benefits

• A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
• Leaders who support your development through coaching and managing opportunities
• Ability to make a difference and lasting impact
• Work in a dynamic, collaborative, progressive, and high-performing team
• A world-class training program in financial services
• Flexible work/life balance options
• Opportunities to do challenging work

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level
• Education Level: No Education Listed