Principal AI Engineer

Scotiabank•Dallas, TX

About The Position

Scotiabank is seeking a Principal Engineer to lead the discovery, architecture, and delivery of AI and agentic solutions across Canadian Banking, International Banking, Global Banking and Markets, Global Wealth Management, Fraud, and Anti-Money Laundering. This hands-on technical leadership role spans data engineering, AI engineering, software engineering, and MLOps. The position partners with business leaders to identify opportunities, translate complex workflows into practical solutions, and guide multidisciplinary teams from experimentation through production deployment and ongoing operation. It requires sound engineering judgment to select among traditional automation, machine learning, generative AI, and agentic workflows, while meeting enterprise expectations for security, reliability, explainability, and measurable business value. The Principal AI Engineer partners with business stakeholders to discover and prioritize use cases, then leads end-to-end technical architecture and delivery across data ingestion, processing, model development, retrieval, agent orchestration, application integration, and production operations. The role develops reusable engineering patterns, builds critical components, leads reviews, resolves complex technical challenges, and mentors engineers. It also establishes data and AI engineering practices, MLOps and DevOps processes, production monitoring, and governance controls. The role is based in Dallas, Texas.

Requirements

Extensive experience in software, data, or machine learning engineering, with technical leadership across complex enterprise initiatives.
Ability to take AI or machine learning solutions from business discovery through production deployment and operation.
Strong hands-on proficiency in Python, SQL, API design, distributed systems, and modern software engineering practices.
Deep experience with Databricks, Apache Spark, Delta Lake, and enterprise data pipeline architecture.
Practical experience building generative AI and agentic applications, including retrieval, tool integration, orchestration, and evaluation.
Experience with MLflow and MLOps, including experiment tracking, model registration, deployment, monitoring, and lifecycle governance.
Strong knowledge of Azure architecture, identity, networking, security, and cloud-native application delivery.
Experience with Docker, Kubernetes, CI/CD pipelines, automated testing, and infrastructure as code.
Ability to communicate technical trade-offs to executives, business stakeholders, architects, and engineering teams.
Ability to lead through influence and raise engineering standards across organizational boundaries.

Nice To Haves

Experience in banking or financial services, particularly banking operations, capital markets, wealth management, fraud detection, or AML.
Experience delivering AI solutions in regulated environments and collaborating with model risk and governance functions.
Familiarity with Unity Catalog, enterprise metadata platforms, semantic models, vector search, and knowledge graphs.
Experience with event-driven architectures, secure agent tool interfaces, and Model Context Protocol (MCP).
Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Responsibilities

Partner with line-of-business stakeholders, product owners, and domain experts to understand business processes, operational challenges, and AI opportunities.
Lead discovery workshops and prioritize AI and agentic use cases based on business value, feasibility, data readiness, risk, and delivery effort.
Translate opportunities into solution designs, delivery roadmaps, technical backlogs, and measurable acceptance criteria.
Assess where autonomous execution is appropriate and where deterministic controls, human review, or approval are needed.
Own end-to-end technical architecture across data ingestion, processing, model development, retrieval, agent orchestration, application integration, and production operations.
Design agentic workflows using tool integrations, enterprise APIs, retrieval-augmented generation, state management, and human oversight.
Establish reusable engineering patterns, reference architectures, and shared components across business lines.
Lead architecture and code reviews, resolve complex engineering challenges, and guide teams through technical trade-offs.
Build critical components and reference implementations; mentor engineers and influence technical decisions across teams.
Design scalable batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, SQL, Python, and Azure services.
Establish data quality, lineage, metadata, access controls, and curated data products for analytics, machine learning, and AI applications.
Develop and integrate machine learning models, LLM-powered applications, retrieval services, and agentic workflows into enterprise systems.
Define evaluation approaches for predictive performance, retrieval quality, response accuracy, tool execution, and end-to-end task completion.
Establish lifecycle practices for experiments, models, prompts, agents, and evaluation datasets using MLflow and supporting tools.
Implement automated testing, CI/CD, infrastructure as code, deployment controls, rollback strategies, and environment management.
Design and deploy containerized services using Docker and Kubernetes, including Azure Kubernetes Service.
Define production monitoring for data and model drift, AI quality, latency, availability, cost, and operational failures.
Partner with platform engineering, cybersecurity, and operations teams to deliver resilient services with clear ownership, support models, and service-level objectives.
Embed privacy, security, auditability, explainability, and responsible AI requirements into solution design and delivery.
Partner with model risk management, compliance, data governance, and cybersecurity teams throughout the delivery lifecycle.
Implement controls for sensitive data access, prompt injection, unsafe tool execution, and unintended agent actions.
Ensure production solutions maintain traceable decisions, evaluation evidence, and appropriate human accountability.