Engineer

Scotiabank•Dallas, TX

About The Position

Scotiabank is seeking a Senior AI Engineer to design, build, and operate AI and agentic solutions supporting Canadian Banking, International Banking, Global Banking and Markets, Global Wealth Management, Fraud, and Anti-Money Laundering. This hands-on engineering role spans AI applications, data pipelines, software services, and MLOps. The engineer will work with business stakeholders, product owners, data scientists, and platform teams to turn business workflows into secure, reliable production solutions using Databricks, MLflow, Kubernetes, and Azure-native technologies. The role owns the design and delivery of assigned components and workflows, contributes to technical decisions with Principal Engineers and architects, and supports solutions throughout their production lifecycle. It requires judgment in selecting deterministic automation, machine learning, generative AI, or agentic approaches, and includes engineering, evaluation, production delivery, security, responsible AI, and collaboration responsibilities.

Requirements

Strong professional experience in software, data, or machine learning engineering, including hands-on delivery of production AI or machine learning solutions.
Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
Hands-on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
Experience with MLflow and MLOps practices such as experiment tracking, model registration, versioning, deployment, and monitoring.
Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, including the ability to deploy and troubleshoot services.
Ability to independently deliver well-defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
Understanding of AI evaluation, data privacy, secure tool use, human oversight, and the limitations of probabilistic model outputs.

Nice To Haves

Experience in financial services, particularly banking, capital markets, wealth management, fraud detection, or AML.
Experience implementing retrieval and vector search, document processing, semantic models, or knowledge graph integrations.
Familiarity with Model Context Protocol (MCP), event-driven architectures, and agent orchestration frameworks.
Experience working in regulated environments with model risk, audit, and data governance requirements.
Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Responsibilities

Participate in business discovery to understand workflows, data sources, pain points, and opportunities for AI-enabled improvement.
Translate requirements into technical designs, implementation tasks, estimates, and testable acceptance criteria with product owners and technical leads.
Assess data readiness, integration dependencies, and technical feasibility, and build prototypes to validate assumptions.
Define suitable boundaries for agent autonomy, human review, approvals, and deterministic business rules.
Build LLM-powered applications and agentic workflows using tool calling, enterprise APIs, retrieval-augmented generation, structured outputs, and state management.
Implement document ingestion, parsing, chunking, embeddings, indexing, retrieval, and reranking to provide relevant, permission-aware context.
Integrate predictive models and business rules into AI workflows in collaboration with data scientists and domain experts.
Implement validation, timeouts, retries, idempotency, and fallback paths to make workflows dependable and prevent unintended repeated actions.
Compare models and orchestration options against task quality, latency, cost, and operational requirements.
Develop and maintain batch and streaming data pipelines using Azure Databricks, Apache Spark, Delta Lake, Python, and SQL.
Implement data quality checks, schema validation, lineage, and access controls using enterprise standards and Unity Catalog where applicable.
Build tested APIs and services connecting AI applications to enterprise data, business systems, and approved tools.
Write maintainable code with clear interfaces, unit and integration tests, documentation, and peer review; troubleshoot issues across application and data layers.
Use MLflow and supporting tools to track experiments, register models, version prompts and configurations, and retain evaluation results and deployment evidence.
Build representative evaluation datasets and automated checks for model performance, retrieval relevance, grounded responses, tool execution, and end-to-end task completion.
Implement CI/CD pipelines, automated regression testing, environment configuration, and controlled releases with rollback support.
Containerize services with Docker and deploy them to Kubernetes, including AKS, using approved infrastructure-as-code patterns.
Instrument applications with tracing, logs, and metrics; monitor AI quality, data and model drift where applicable, latency, availability, and consumption costs.
Investigate production incidents, perform root-cause analysis, maintain runbooks, and contribute to operational support under the team’s agreed support model.
Implement least-privilege access, secure secrets management, data protection, and audit logging using enterprise security and Azure identity standards.
Apply safeguards against prompt injection, sensitive data disclosure, unsafe tool execution, and unauthorized agent actions.
Implement human approval and escalation for sensitive decisions or actions, preserving traceability of inputs, outputs, and tool activity.
Work with cybersecurity, model risk, compliance, and data governance partners to address controls and remediation items.
Align component designs with enterprise architecture and reusable platform capabilities in collaboration with Principal Engineers and architects.
Collaborate with data engineers, data scientists, platform engineers, and business teams to resolve dependencies and deliver integrated solutions.
Mentor engineers through code reviews, pairing, and knowledge sharing; contribute reusable components and practical engineering guidance.
Communicate progress, blockers, technical trade-offs, and delivery risks clearly and early.