Data Engineer

Ontario Securities Commission•Toronto, ON

10d

About The Position

Under the guidance of the Technical Manager, Data & Analytics, this position will be responsible for partnering with internal stakeholders to understand and assess current and future business needs to inform design and development of data solutions, build data pipelines, and perform data management and optimization. The candidate will have good knowledge of capital markets data sets. This role will be required to ensure the capturing and translation of complex business logic into technical data deliverables. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up and support business units, data analysts and data scientists on data initiatives and ensure optimal data delivery. They will need to work to continuously improve and redeploy data solutions to meet evolving regulatory needs.

Requirements

10+ years of experience working with data in roles such as Data Engineer, ETL Engineer, or similar
Graduate degree in Computer Science, Statistics, Informatics, Information Systems, or another quantitative field
Cloud & Data Platforms
Strong hands-on experience with Databricks (mandatory)
Experience with Azure Data Factory for orchestration and data pipelines
Experience with Azure Synapse Analytics for data warehousing and analytics
Strong understanding of Azure cloud-based data engineering ecosystems
Data Engineering & Architecture
Experience building and optimizing large-scale data pipelines and datasets
Experience designing and managing batch and streaming data pipelines
Experience managing metadata, dependencies, workload orchestration, and data structures
Proven ability to work with large, disconnected datasets and extract meaningful business value
Data Ingestion & Processing
Strong hands-on experience with Databricks Auto Loader for scalable incremental ingestion
Experience building production-grade pipelines using Delta Live Tables (DLT) / declarative pipelines
Experience with schema evolution, data ingestion frameworks, and pipeline optimization
Data Governance, Contracts & Frameworks
Strong experience designing and implementing YAML-based data contracts
Hands-on experience enforcing schema consistency, validation rules, and data quality standards
Experience building metadata-driven frameworks where pipeline behavior is configuration-driven (no hardcoding)
Strong experience with data governance, lineage, and validation frameworks
Programming & Data Skills
Advanced working knowledge of SQL and relational databases, including query authoring and optimization
Strong programming skills in Python and PySpark (mandatory)
Exposure to R, Java, and scripting languages such as PowerShell
Strong analytical skills working with both structured and unstructured datasets
Strong ability to collect, organize, analyze, and interpret large volumes of data with high accuracy and attention to detail
Reusable Engineering & Packaging
Strong experience building and deploying reusable Python packages (Wheel files) in Databricks environments
Experience building modular frameworks for: ETL utilities, Data validation, Logging and auditing, Shared reusable pipeline components
Visualization & Reporting
Strong experience with Databricks Dashboards (mandatory)
Experience with Power BI for enterprise reporting and visualization
CI/CD, DevOps & Testing
Strong experience with Azure DevOps (CI/CD pipelines mandatory)
Experience with Databricks Asset Bundles (DAB) for deployment automation
Experience validating data quality, schema evolution, and regression testing
Experience supporting production releases and workflow automation
Exposure to unit test automation for data pipelines
Analytical & Problem-Solving Skills
Extensive experience performing root cause analysis (RCA) on data and process issues
Experience designing processes supporting: Data transformation, Metadata management, Dependency management, Workload orchestration
Strong ability to identify opportunities for process and data improvement
Soft Skills & Mindset
Experience working with cross-functional teams in dynamic environments
Strong curiosity and interest in emerging technologies
Ability to adapt to change and support teams through transformation journeys
Strong communication, collaboration, and stakeholder engagement skills

Responsibilities

Partners with internal stakeholders across the organization to understand current and future business needs, to translate and develop overall strategy for data platform architecture.
As familiarity on projects increases, work to proactively address needs with the business, and validate assumptions with business partners
Independently provides recommendations to internal stakeholders on data solutions and assesses impact and risks for branch and broader organization implications, in collaboration with Business Data Architect and the Data Management and Reporting Lead
Partner with business units to translate and assess business requirements into data ingestion and standardization scripts.
Determine appropriate solution, assess impact and risk of solutions, develop and maintains optimal yet scalable data pipelines that supports efficient extraction, transformation, and loading of data from a wide variety of high volume and complex data sources
Work with Security Specialists to ensure compliance with security and data management requirements
As subject matter expert, provide advice in the build of analytics tools that utilize the data pipeline to provide actionable insights and inform of risks and impact to ensure successful implementation.
As subject matter expert, advise analytics and business teams to develop data models and constantly strive for excellence in our data capabilities
Recommend and implement processes and systems to continuously monitor data quality to ensure that production data is always accurate and available for authorized stakeholders
Lead data analysis to troubleshoot data related issues and provide expertise in the resolution of data issues
As subject matter expert in data engineering, partner with Security, Architecture and Technical Services to align solution designs with requirements
Act as subject matter expert in agile pods (multi-disciplinary teams) to complete complex projects, representing the data engineering function.
Ensure quality of data is upheld throughout the transformation process and meets the desired state within the target data architecture
Contribute towards data integrations and data quality framework to ensure that OSC data engineering practices are following standards set by our Data Governance teams, and limit the access and processing of data as per regulations and internal controls
In partnership with business, Digital Solutions and IT branches increase data accessibility and fostering a data-driven decision-making culture across the organization
As the subject matter expert, provides regular guidance and advice in management and use of data to business units in the Spokes and across the Digital Solutions branch.
Collaborate with the data strategy chapter as data engineering expert in - (a) identifying and acquiring data from internal and external data sources, (b) Prioritizing business and information needs
Support interpretation and analysis of data and trends using programming or statistical techniques and generate insights and reports on on-going basis