Principal AI Engineer

Scotiabank•Dallas, TX

About The Position

Scotiabank is seeking a Principal Engineer to lead discovery, architecture, and delivery of AI and agentic solutions across Canadian Banking, International Banking, Global Banking and Markets, Global Wealth Management, Fraud, and Anti-Money Laundering. This hands-on technical leadership role spans data engineering, AI engineering, software engineering, and MLOps. The role partners with business leaders to identify opportunities, turn complex workflows into practical solutions, and guide multidisciplinary teams from experimentation through production deployment and ongoing operation. It requires sound engineering judgment in selecting traditional automation, machine learning, generative AI, or agentic workflows, and delivering solutions that meet enterprise requirements for security, reliability, explainability, and measurable business value. The Principal AI Engineer partners with stakeholders to discover and prioritize use cases, then owns end-to-end technical architecture and guides delivery. The position includes building data pipelines and AI applications, establishing engineering and MLOps practices, deploying and monitoring production services, and embedding responsible AI, security, privacy, auditability, and governance controls. The role also builds critical components, reviews architecture and code, mentors engineers, and influences engineering practices across teams. The technology environment includes Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, Azure data services, Azure-hosted AI services, MLflow, Python, SQL, REST APIs, microservices, Docker, Kubernetes and AKS, Azure Functions, Azure API Management, Azure DevOps or GitHub Actions, Terraform or Bicep, Azure Monitor, Application Insights, Key Vault, and Microsoft Entra ID. The role is based in Dallas, Texas. Scotiabank describes its benefit programs as flexible and designed to support employees’ family, financial, physical, mental, and social health needs.

Requirements

Extensive experience in software, data, or machine learning engineering, with demonstrated technical leadership across complex enterprise initiatives.
Proven ability to take AI or machine learning solutions from business discovery through production deployment and operation.
Strong hands-on proficiency in Python, SQL, API design, distributed systems, and modern software engineering practices.
Deep experience with Databricks, Apache Spark, Delta Lake, and enterprise data pipeline architecture.
Practical experience building generative AI and agentic applications, including retrieval, tool integration, orchestration, and evaluation.
Experience with MLflow and MLOps practices, including experiment tracking, model registration, deployment, monitoring, and lifecycle governance.
Strong knowledge of Azure architecture, identity, networking, security, and cloud-native application delivery.
Experience with Docker, Kubernetes, CI/CD pipelines, automated testing, and infrastructure as code.
Ability to communicate technical trade-offs clearly to executives, business stakeholders, architects, and engineering teams.
Demonstrated ability to lead through influence and raise engineering standards across organizational boundaries.

Nice To Haves

Experience in banking or financial services, particularly banking operations, capital markets, wealth management, fraud detection, or AML.
Experience delivering AI solutions within regulated environments and collaborating with model risk and governance functions.
Familiarity with Unity Catalog, enterprise metadata platforms, semantic models, vector search, and knowledge graphs.
Experience with event-driven architectures, secure agent tool interfaces, and Model Context Protocol (MCP).
Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Responsibilities

Partner with line-of-business stakeholders, product owners, and domain experts to understand processes, operational challenges, and AI opportunities.
Lead discovery workshops and prioritize AI and agentic use cases based on business value, technical feasibility, data readiness, risk, and delivery effort.
Translate opportunities into solution designs, delivery roadmaps, technical backlogs, and measurable acceptance criteria.
Assess when autonomous execution is appropriate and when deterministic controls, human review, or approval are required.
Own end-to-end technical architecture spanning data ingestion and processing, model development, retrieval, agent orchestration, application integration, and production operations.
Design agentic workflows with tool use, enterprise APIs, retrieval-augmented generation, state management, and human oversight.
Establish reusable engineering patterns, reference architectures, and shared components across business lines.
Lead architecture and code reviews, resolve complex engineering challenges, and guide teams through technical trade-offs.
Build critical components and reference implementations, mentor engineers, and influence technical decisions across teams.
Design scalable batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, SQL, Python, and Azure-native services.
Establish data quality, lineage, metadata, access controls, and curated data products for analytics, machine learning, and AI applications.
Develop and integrate machine learning models, LLM-powered applications, retrieval services, and agentic workflows into enterprise systems.
Define evaluations for predictive performance, retrieval quality, response accuracy, tool execution, and end-to-end task completion.
Establish lifecycle practices for experiments, models, prompts, agents, and evaluation datasets using MLflow and supporting tools.
Implement automated testing, CI/CD, infrastructure as code, deployment controls, rollback strategies, and environment management.
Design and deploy containerized services using Docker and Kubernetes, including Azure Kubernetes Service.
Define production monitoring for data and model drift, AI quality, latency, availability, cost, and operational failures.
Partner with platform engineering, cybersecurity, and operations teams to deliver resilient services with clear ownership, support models, and service-level objectives.
Embed privacy, security, auditability, explainability, and responsible AI requirements into solution design and delivery.
Partner with model risk management, compliance, data governance, and cybersecurity teams throughout delivery.
Implement controls for sensitive data access, prompt injection, unsafe tool execution, and unintended agent actions.
Ensure production solutions maintain traceable decisions, evaluation evidence, and appropriate human accountability.