Principal Data Engineer

Scotiabank•Dallas, TX

2d•Onsite

About The Position

Principal Data Engineer will lead the design and evolution of enterprise-scale, multi-cloud data platforms and ingestion capabilities. This role is responsible for defining technical architecture, establishing engineering standards, and solving complex data integration challenges across cloud ecosystems including Azure, Google Cloud Platform (GCP), AWS, and modern Lakehouse platforms. As a senior technical leader, you will drive the design of resilient, high-volume data ingestion frameworks, establish reusable patterns for onboarding diverse data sources, and provide hands-on leadership to engineering teams. You will partner closely with Enterprise Architects, Solution Architects, Product Owners, Data Scientists, Business Systems Analysts, Security, and Platform Engineering teams to deliver scalable, secure, and cost-efficient data solutions that enable enterprise AI, analytics, and regulatory initiatives.

Requirements

10+ years of experience designing and delivering large-scale enterprise data platforms.
Proven experience architecting cloud-native and hybrid data solutions across Azure, AWS, and/or GCP.
Deep expertise designing enterprise ingestion architectures supporting high-volume, high-velocity, and mission-critical data workloads.
Experience establishing technical roadmaps, reference architectures, engineering standards, and platform operating models.
Strong ability to lead complex technical discussions and influence senior stakeholders across business and technology organizations.
Expert-level experience building resilient, scalable ETL/ELT, CDC, event-driven, streaming, and batch ingestion pipelines.
Extensive experience integrating data from enterprise applications, SaaS platforms, APIs, relational databases, NoSQL databases, messaging platforms, data warehouses, and external partner systems.
Strong understanding of data modeling, schema evolution, metadata management, data lineage, and data governance principles.
Experience implementing end-to-end data quality frameworks, reconciliation processes, and operational controls.
Deep expertise working with structured, semi-structured, and unstructured datasets at enterprise scale.
Expertise with modern Lakehouse architectures, including Databricks and Delta Lake.
Strong experience with cloud-native data services across Azure, AWS, and GCP.
Advanced programming skills in Python, SQL, Scala, and Java.
Experience with distributed processing frameworks and large-scale data processing technologies.
Experience with event-streaming and messaging technologies such as Kafka, Event Hubs, Pub/Sub, or equivalent platforms.
Strong understanding of containerization, orchestration, microservices, and cloud-native architecture patterns.
Extensive experience implementing CI/CD pipelines using GitHub, Bitbucket, Azure DevOps, GitLab, Terraform, and Infrastructure-as-Code practices.
Strong experience embedding security controls, compliance requirements, and automated testing into engineering pipelines.
Demonstrated expertise in troubleshooting, root-cause analysis, performance optimization, and production support.
Ability to translate complex technical architectures into clear business outcomes for executive and non-technical audiences.
Proven track record of leading cross-functional teams and delivering complex data programs in large enterprise environments.
Strong mentorship and coaching capabilities with a passion for developing engineering talent.

Nice To Haves

5+ years of experience with Databricks and Lakehouse architecture.
Experience supporting enterprise AI, machine learning, analytics, or real-time decisioning platforms.
Experience designing multi-region, highly available, and disaster-resilient data platforms.
Knowledge of enterprise governance, regulatory, privacy, and security requirements within highly regulated industries.

Responsibilities

Architect and lead the implementation of enterprise-scale data ingestion frameworks and patterns capable of processing structured, semi-structured, streaming, and unstructured data across hybrid and multi-cloud environments.
Define and implement reusable ingestion frameworks, reference architectures, and engineering standards that accelerate source onboarding and reduce delivery complexity.
Lead the technical design and optimization of high-throughput batch, streaming, CDC (Change Data Capture), event-driven, and file-based integration patterns.
Design and implement data pipelines for mission-critical workloads, ensuring scalability, resiliency, observability, security, and operational excellence.
Solve complex data integration challenges involving SaaS platforms, databases, mainframes, data warehouses, real-time messaging systems, and third-party applications.
Establish robust data quality, lineage, metadata, reconciliation, and governance controls throughout the ingestion lifecycle.
Drive adoption of modern engineering practices including Infrastructure-as-Code, CI/CD, automated testing, GitOps, and DevSecOps.
Define and implement platform observability strategies including monitoring, logging, alerting, performance tuning, and operational support models.
Provide technical leadership, architectural guidance, and mentorship to engineering teams while driving engineering excellence across multiple initiatives.
Evaluate emerging technologies and recommend strategies to improve scalability, reliability, performance, security, and cost optimization of enterprise data platforms.