ETL Lead

Delan Associates•Greenville, SC

5d•Onsite

About The Position

The ETL Team Lead owns end-to-end operational support for Canal’s existing data stack, ensuring the success of day-to-day production operations. This role involves monitoring daily ETL loads, troubleshooting pipeline failures, performance issues, and schema mismatches, and performing root-cause analysis. The lead will ensure SLA adherence and on-time delivery of critical reporting datasets, provide direction for ETL maintenance, and refactor or retire outdated processes. The position requires maintaining and improving existing pipelines using various technologies. Additionally, the role involves working with operational teams to enhance runbooks, SOPs, and monitoring dashboards, partnering with IT operational segments and business SMEs to minimize downtime, and implementing proactive measures for daily processing. The ETL Tech Lead will also support business continuity efforts, including development support coverage for critical data pipelines and support for month-end and quarter-end financial reporting cycles. This role serves as the steady-state technical owner of the data operations layer during the Canal modernization journey. The lead will guide onshore/offshore developers, review code, enforce best practices, and mentor junior engineers. Collaboration with Scrum Masters, Project Managers, Enterprise Architecture, QA Automation, Change Management, and AMS support teams is essential. Key development areas include creating reusable ingestion patterns, modernizing ETL workloads using Delta Lake and Medallion Architecture, and building scalable data ingestion pipelines with Azure Data Factory, MS Fabric, Databricks, and Synapse Pipelines. Experience with real-time, streaming, and event-driven engineering using Event Hub, Fabric Real-Time Analytics, Databricks Structured Streaming, and KQL is crucial for developing operational insights and automation capabilities. The role also involves leading the strategy, design, and engineering of Canal’s modern Azure data ecosystem, implementing Medallion Architecture, leveraging Delta tables, building ingestion and transformation pipelines, and enabling real-time analytics. Data modeling, curation, and governance are key, including developing analytics-ready datasets, assisting with Data Governance tools, and establishing data quality frameworks. Optional responsibilities include preparing ML-ready datasets for various business use cases.

Requirements

ETL Team Lead owns end-to-end operational support for Canal’s existing data stack.
Ensure success of day-to-day production Operations.
Monitor daily ETL loads across SQL jobs, DHIC (GW Data Hub and Info Center) and legacy SSIS packages.
Work with AMS team where necessary to troubleshoot pipeline failures, performance issues, schema mismatches, permissions issues, and cloud resource failures.
Work with AMS team where necessary to perform root-cause analysis and to implement permanent fixes.
Ensure SLA adherence and on-time delivery of critical reporting datasets for scheduled ETL jobs.
Provide direction for both AMS and ETL Developers for Legacy & Current ETL maintenance.
Refactor or retire outdated or redundant ETL processes.
Maintain and improve existing pipelines that utilize the following technologies: Microsoft SQL Server Database programming T-SQL Scripting SQL Server Integration Services Microsoft PowerShell Guidewire Data Hub and Info Center Oracle Database programming Oracle PL-SQL Scripting SAP BODS (SAP BusinessObjects Data Services) PostgreSQL Scripting
Work with AMS team to assist with the creation and/or enhancement of operational runbooks, SOPs, monitoring dashboards, and incident response workflows.
Partner with other IT operational segments, business SMEs, and AMS team to minimize downtime and to ensure that business SLAs are met.
Improve upon existing, as well as implement new proactive for daily processing.
Work with AMS team to ensure development support coverage for critical data pipelines (rotation-based).
Support month-end and quarter-end financial reporting cycles.
Coordinate production releases and validate deployments.
Serve as technical lead guiding onshore/offshore developers.
Review code, enforce best practices, and mentor junior engineers.
Partner with Scrum Masters, Project Mangers, Enterprise Architecture, QA Automation, Change Management, and AMS support teams.
Develop reusable ingestion patterns for Guidewire DataHub and InfoCenter, HubSpot, Telematics, and other facets of the business.
Work with Canal Architects to modernize existing ETL workloads using Delta Lake, Medallion Architecture, and Fabric Lakehouse.
Build scalable data ingestion pipelines using potential upcoming technology (Azure Data Factory, MS Fabric , Databricks, Synapse Pipelines, etc.).
Work to bring in Internal and External integrations data into the platform.
Proven experience designing and implementing real-time data pipelines using Event Hub, Fabric Real-Time Analytics, Databricks Structured Streaming, and KQL-based event processing.
Develop and enable real-time operational insights and automation capabilities—including Telematics alerting, FNOL automation, and fraud/VNOS/VNOP detection—through event-driven architectures and streaming analytics.
Lead the strategy, design, and engineering of Canal’s modern Azure data ecosystem using next-generation tools and Medallion Architecture, including:
Implementing Medallion Architecture (Bronze/Silver/Gold) across Fabric Lakehouse, Warehouse, Eventhouse, and KQL Database.
Leveraging Delta tables with schema enforcement, ACID compliance, and versioning.
Building ingestion and transformation pipelines using Dataflows Gen2, Fabric Pipelines, and Notebook-based ETL.
Enabling real-time analytics using RTA, EventStreams, and Event Hub.
Driving notebook-based transformations and advanced engineering using Databricks and Synapse runtimes.
Develop curated, analytics-ready datasets to support Power BI, operational reporting, and advanced analytics use cases.
Assist Canal architect with implementation of Data Governance tools.
Establish robust data quality, validation, alerting, and observability frameworks to maintain accuracy, consistency, and trust in enterprise data.
Prepare ML-ready datasets for pricing, risk, fraud detection, underwriting, claims leakage, and predictive insights (Optional).

Nice To Haves

Insurance Domain

Responsibilities

Own end-to-end operational support for Canal’s existing data stack.
Ensure success of day-to-day production Operations.
Monitor daily ETL loads across SQL jobs, DHIC (GW Data Hub and Info Center) and legacy SSIS packages.
Work with AMS team where necessary to troubleshoot pipeline failures, performance issues, schema mismatches, permissions issues, and cloud resource failures.
Work with AMS team where necessary to perform root-cause analysis and to implement permanent fixes.
Ensure SLA adherence and on-time delivery of critical reporting datasets for scheduled ETL jobs.
Provide direction for both AMS and ETL Developers for Legacy & Current ETL maintenance.
Refactor or retire outdated or redundant ETL processes.
Maintain and improve existing pipelines.
Work with AMS team to assist with the creation and/or enhancement of operational runbooks, SOPs, monitoring dashboards, and incident response workflows.
Partner with other IT operational segments, business SMEs, and AMS team to minimize downtime and to ensure that business SLAs are met.
Improve upon existing, as well as implement new proactive for daily processing.
Work with AMS team to ensure development support coverage for critical data pipelines (rotation-based).
Support month-end and quarter-end financial reporting cycles.
Coordinate production releases and validate deployments.
Serve as technical lead guiding onshore/offshore developers.
Review code, enforce best practices, and mentor junior engineers.
Partner with Scrum Masters, Project Mangers, Enterprise Architecture, QA Automation, Change Management, and AMS support teams.
Develop reusable ingestion patterns for Guidewire DataHub and InfoCenter, HubSpot, Telematics, and other facets of the business.
Work with Canal Architects to modernize existing ETL workloads using Delta Lake, Medallion Architecture, and Fabric Lakehouse.
Build scalable data ingestion pipelines using potential upcoming technology (Azure Data Factory, MS Fabric , Databricks, Synapse Pipelines, etc.).
Work to bring in Internal and External integrations data into the platform.
Develop and enable real-time operational insights and automation capabilities—including Telematics alerting, FNOL automation, and fraud/VNOS/VNOP detection—through event-driven architectures and streaming analytics.
Lead the strategy, design, and engineering of Canal’s modern Azure data ecosystem using next-generation tools and Medallion Architecture.
Implement Medallion Architecture (Bronze/Silver/Gold) across Fabric Lakehouse, Warehouse, Eventhouse, and KQL Database.
Leverage Delta tables with schema enforcement, ACID compliance, and versioning.
Build ingestion and transformation pipelines using Dataflows Gen2, Fabric Pipelines, and Notebook-based ETL.
Enable real-time analytics using RTA, EventStreams, and Event Hub.
Drive notebook-based transformations and advanced engineering using Databricks and Synapse runtimes.
Develop curated, analytics-ready datasets to support Power BI, operational reporting, and advanced analytics use cases.
Assist Canal architect with implementation of Data Governance tools.
Establish robust data quality, validation, alerting, and observability frameworks to maintain accuracy, consistency, and trust in enterprise data.
Prepare ML-ready datasets for pricing, risk, fraud detection, underwriting, claims leakage, and predictive insights (Optional).