Engineer II – Senior Data Engineer (Azure Databricks / Data Engineering)

TD•Toronto, ON

1d•CA$96,900 - CA$136,800•Onsite

About The Position

Develops and maintains technical solutions that adhere to engineering and architectural design principles while meeting business requirements. Provides deep technical expertise in data engineering with a focus on Azure Databricks, Python notebook development, job orchestration, API service layers, and scalable data platforms across ADLS, ADF, Azure Databricks (including Unity Catalog and Lakeflow), Azure SQL, Airflow, and SQL. Designs, builds, operationalizes, and supports reliable, secure, and high-performing enterprise-grade data solutions that can scale to process multi-millions of records daily in compliance with enterprise and industry standards.

Requirements

University or post-graduate degree
Strong academic background (e.g., computer science, engineering)
7 + years relevant experience in data engineering, including Azure Databricks, Python notebooks, job orchestration, SQL development, API service layer implementation, and hands-on experience with ADLS, ADF, Azure Databricks (with Unity Catalog and Lakeflow), Azure SQL, and Airflow

Nice To Haves

experience integrating AI into the SDLC using reusable prompts, skills, instructions, and agents
building enterprise-grade software and frameworks
designing scalable solutions capable of handling multi-millions of records daily

Responsibilities

Leverage deep technology expertise in data engineering, Azure Databricks, Python, SQL, and cloud data platforms for own area of specialization to deliver and ensure that all areas across the organization that provision, manage and support various technologies have the necessary tools, processes and documentation required to effectively execute on their respective mandates
Execute on Engineering strategy as it relates to the introduction of tools and the automation of data ingestion, transformation, notebook execution, job scheduling, and deployment activities across Azure Databricks, ADF, Airflow, API services, and Azure-based data platforms
Partner with the Operations team to automatically integrate with appropriate tools and processes as part of automated/self-serve data pipeline, notebook job, and API service releases
Work with partners across Technology and apply in-depth understanding of relevant business data needs to identify and leverage synergies across the various areas
Act as the expert or lead innovator and agent of change for the programs and services under management, driving modern data engineering practices, reusable frameworks, and platform standards
Work with other teams to implement best practices for engineering and management, including enterprise-grade software development, reusable engineering frameworks, and AI integration into the SDLC through reusable prompts, skills, instructions, and agents
Work with vendor platform providers and engineering peers to keep abreast of trends, products, frameworks, and applications
Identify and effectively manage stakeholder engagement and impacts across the enterprise
Interpret client needs, assess engineering related requirements and identify solutions to non-standard requests
Apply best practices and knowledge of internal / external business issues to improve products or services in own discipline
Monitor and control costs within own work
May interact with governance and control groups, (e.g. regulatory / operational risk, compliance and audit) to provide subject matter expertise and consult on risk issues / items related to Engineering technology and tools
May develop and/or contribute to negotiations of third party contracts/agreements
Maintain knowledge and understanding of external development, engineering and emerging solutions, market conditions and their impact, with particular focus on cloud data engineering, Databricks capabilities, API enablement, and scalable data processing patterns
Proactively identify emerging technologies and innovative solutions for building more robust platform domains, including data platforms capable of handling multi-millions of records daily with strong reliability, governance, and performance
Continuously enhance knowledge/expertise in own area and keep current with emerging industry trends, new technologies and best practices in the external market that can contribute to delivering effective client solutions across Azure Databricks, ADLS, ADF, Azure SQL, Airflow, SQL, API service development, and modern data engineering tooling
Prioritize and manage own workload in order to deliver quality results and meet timelines
Support a positive work environment that promotes service to the business, quality, innovation and teamwork and ensure timely communication of issues/ points of interest
Participate in knowledge transfer with senior management, the team, other technical areas and business units
Work effectively as a team, supporting other members of the team in achieving business objectives and providing client services
Identify and recommend opportunities to enhance productivity, effectiveness and operational efficiency of the business unit and/or team through reusable data engineering components, automated orchestration, AI-assisted SDLC practices, and scalable processing frameworks