Staff Software Engineer

RBC•Calgary, AB

3d•Onsite

About The Position

We're seeking a seasoned Staff Software Engineer to join the RBC Borealis AI Platform team and own the end-to-end lifecycle of machine learning systems—from experimentation and validation through to high-throughput production serving at scale. You'll be the technical anchor for operationalizing vision language models and document processing systems that handle thousands of documents per minute, setting the bar for reliability, observability, and engineering excellence across our AI platform. You'll lead the design and evolution of our scalable document processing platform—a production system that combines event-driven architecture, vision language models, and cloud-native infrastructure to extract intelligence from financial documents at enterprise scale: * ML System Operationalization: Own the production lifecycle of LLM and computer vision models, from integration and validation to serving, monitoring, and continuous improvement at 1000+ documents/minute throughput * Platform Architecture: Design resilient microservices using FastAPI and event-driven patterns with Apache Kafka, ensuring 99.5%+ reliability for mission-critical financial document processing * Scalable Infrastructure: Build and optimize Kubernetes-native workloads with KEDA-based autoscaling (3-50 replicas dynamically), PostgreSQL/MongoDB data layers, and S3 object storage with lifecycle management * Observability & Reliability: Establish comprehensive monitoring, alerting, and SRE practices that provide deep visibility into model performance, system health, and business metrics across distributed services This is a rare opportunity to shape the foundation on which Canada's largest financial institution runs its most critical AI workloads, working directly with leading researchers in machine learning while having access to rich, massive datasets and the computational resources to support groundbreaking innovation.

Requirements

5-8+ years of software engineering experience with 3+ years focused on ML systems, data platforms, or high-scale distributed systems
Deep expertise in Python and production-grade API frameworks (FastAPI, Flask, or similar) with strong software design principles
Proven track record operationalizing ML models in production—you've integrated LLMs, vision models, or similar AI services into scalable systems
Strong hands-on experience with event-driven architectures using Apache Kafka, RabbitMQ, or cloud-native messaging platforms
Production experience with both SQL (PostgreSQL) and NoSQL (MongoDB, DynamoDB) databases, understanding tradeoffs and optimization strategies
Expert-level knowledge of containerization (Docker) and Kubernetes/OpenShift orchestration, including custom resources, operators, and autoscaling

Responsibilities

Architect production ML pipelines that seamlessly integrate vision language models, OCR engines, and document extraction services into scalable, fault-tolerant systems
Drive technical decisions on complex distributed systems challenges involving data consistency, exactly-once processing semantics, and sub-500ms API response times
Collaborate closely with ML researchers to translate cutting-edge models in computer vision, NLP, and reinforcement learning into production-ready services
Set engineering standards for model serving, A/B testing, feature flags, and gradual rollouts that enable safe, data-driven experimentation at scale
Build sophisticated retry mechanisms with exponential backoff, circuit breakers, dead-letter queues, and fallback strategies that ensure system resilience
Implement advanced event-driven patterns across Kafka topics (ingestion, processing, callbacks, DLQ) with precise consumer group management and lag-based autoscaling
Develop reusable frameworks and libraries for async processing, template-based document parsing, and callback orchestration that accelerate team productivity
Lead the evaluation and adoption of emerging AI technologies, ensuring alignment with enterprise security, compliance, and data governance requirements
Partner with data scientists and ML researchers to understand model requirements, performance characteristics, and integration patterns for production deployment
Work with process engineers and business stakeholders to translate financial document processing needs into robust, scalable technical solutions
Foster strong relationships across platform, infrastructure, and security teams to deliver end-to-end capabilities that span multiple domains
Mentor engineers on distributed systems design, event-driven architecture, ML ops best practices, and cloud-native development patterns
Navigate ambiguity in complex technical challenges, from Kafka partition strategies to LLM provider selection to autoscaling configurations
Identify and mitigate architectural risks before they impact production, using techniques like chaos engineering, load testing, and failure mode analysis
Provide clear, data-driven recommendations to engineering leadership on infrastructure investments, technology choices, and platform roadmap priorities
Drive continuous improvement in system performance, cost efficiency, and developer experience through metrics-driven iteration