Software Engineer III - Metadata Management & Data Discovery

TD•Toronto, ON

1d•CA$125,500 - CA$154,000•Onsite

About The Position

We are seeking a highly technical Software Engineer III – Metadata Management & Data Discovery to help lead the design, engineering, and delivery of enterprise metadata harvesting capabilities across cloud, multi-cloud, and legacy on-premises data platforms. This role will be responsible for ideating, designing, building, and operationalizing scalable metadata harvesting solutions that enable automated discovery, cataloging, lineage, classification, and governance of enterprise data assets. The successful candidate will support the Data Discovery and Metadata Harvesting function by building reusable patterns and integrations that harvest structured and semi-structured metadata from platforms such as Databricks Unity Catalog, Azure ADLS, Azure Synapse, Snowflake, AWS, Google Cloud, Oracle, SQL Server, DB2, Teradata, mainframe files, SFTP/network shares, and other enterprise data stores. The role will be central to advancing Data Compass / Collibra as the enterprise metadata system of record and will partner closely with engineering, cloud infrastructure, architecture, security, governance, and data platform teams to ensure metadata is harvested, normalized, enriched, reconciled, and maintained in a trusted and audit-ready manner.

Requirements

Undergraduate degree, Post Graduate degree or Technical Certificate
Strong academic background (e.g., computer science, engineering).
5-7 years relevant experience
Strong hands-on experience designing and building data engineering, metadata management, data catalog, or data governance technology solutions.
Strong programming experience with Python, SQL, and API-based integration patterns.
Deep experience with Databricks, Unity Catalog, Delta Lake, Spark/PySpark, and cloud data platform engineering.
Strong understanding of Azure infrastructure, including ADLS Gen2, Synapse, Azure SQL, Azure Data Factory, Entra ID, service principals, managed identities, Key Vault, private endpoints, VNets, NSGs, and RBAC.
Experience integrating with or engineering around Collibra, Data Compass, Collibra EDGE, Purview, Alation, Informatica, or similar catalog/governance platforms.
Experience with metadata extraction from structured and semi-structured sources, including databases, data lakes, files, schemas, views, tables, columns, tags, and operational metadata.
Experience with cloud data platforms such as Snowflake, AWS, and/or Google Cloud Platform.
Working knowledge of legacy data platforms such as Oracle, SQL Server, DB2, Teradata, mainframe files, flat files, and SFTP/network-share based data stores.
Strong understanding of data governance concepts, including lineage, classification, ownership, stewardship, business glossary, data quality, and metadata lifecycle management.
Experience designing secure, scalable, supportable engineering solutions with CI/CD, Git, logging, monitoring, automated testing, and production support practices.
Ability to work across technical and non-technical stakeholders and translate complex metadata/platform topics into clear execution plans.

Nice To Haves

Graduate degree nice to have

Responsibilities

Lead the technical design and implementation of enterprise metadata harvesting solutions across cloud, multi-cloud, and on-premises platforms.
Build automated harvesting pipelines and connectors to capture technical, operational, business, lineage, classification, ownership, and stewardship metadata.
Design reusable ingestion patterns using Collibra EDGE connectors, REST APIs, JDBC/ODBC, event-driven approaches, file-based ingestion, and custom adapters where needed.
Engineer metadata harvesting solutions for Databricks Unity Catalog, including catalog, schema, table, view, column, permission, tag, lineage, and operational metadata extraction.
Develop scalable harvesting patterns for Azure data platforms, including ADLS Gen2, Azure SQL, Synapse, Azure Data Factory, Key Vault, private endpoints, service principals, and secure connectivity.
Extend metadata harvesting capabilities across Snowflake, AWS, Google Cloud, and major on-premises data stores.
Design and implement reconciliation controls to compare source platform inventories against harvested assets in Data Compass / Collibra.
Define and maintain asset mapping, attribute mapping, classification rules, lineage stitching logic, enrichment patterns, and metadata quality checks.
Partner with cloud infrastructure, IAM, security, and networking teams to establish secure access patterns for harvesting across environments.
Support environment promotion from DEV/SIT/PAT/PROD with proper controls, monitoring, alerting, logging, and operational readiness.
Build monitoring and support capabilities for harvesting jobs, including failure alerts, retry logic, health checks, lineage validation, and incremental harvesting.
Collaborate with governance and business teams to ensure metadata is complete, trusted, actionable, and aligned to enterprise data standards.
Create technical documentation, architecture diagrams, implementation guides, runbooks, and reusable patterns for onboarding new sources.
Mentor engineers and analysts on metadata harvesting engineering practices, source onboarding, validation, and automation.
Help advance the broader Data Discovery roadmap, including Unity Catalog harvesting, Data Compass integration, lineage expansion, AI/ML metadata harvesting, and critical system metadata coverage.