Lead Data Engineer

The Joint Commission•Oakbrook Terrace, IL

20h

About The Position

Overview Drives the design, development, and optimization of enterprise data pipelines and platforms. Focused on building robust, scalable, and secure data solutions using modern cloud technologies. Mentor engineers, collaborate with analytics and business teams, and ensure data engineering best practices are followed to support analytics, reporting, and operational needs. Responsibilities Data Engineering & Pipeline Development Lead the design, development, and maintenance of scalable data pipelines using Azure Data Factory, Databricks, Foundry, and Fabric. Build and optimize data ingestion, transformation, and integration processes for diverse data sources. Develop and manage data lakes, lakehouses, and data warehouses, primarily leveraging MSSQL and cloud-native platforms. Implement ETL/ELT best practices, ensuring high performance, reliability, and data quality. Automate data workflows and monitor pipeline health, troubleshooting issues as needed. Team Leadership & Collaboration Mentor and guide a team of data engineers, fostering technical growth and collaboration. Work closely with analytics, BI, and business teams to translate requirements into technical solutions. Promote a culture of innovation, continuous improvement, and knowledge sharing within the team. Technology Enablement & Best Practices Evaluate and implement new data engineering tools and technologies to improve efficiency and scalability. Establish and enforce standards for code quality, testing, and documentation in data engineering projects. Champion data quality, reliability, and governance within engineering workflows. Business Intelligence & Analytics Support Enable self-service analytics by building reusable data models and accessible data sets. Collaborate with BI teams to ensure data pipelines meet reporting and dashboard requirements. Translate business requirements into scalable technical solutions. Compliance & Security Ensure data engineering solutions comply with privacy regulations (e.g., GDPR, HIPAA) and internal security standards. Monitor and enforce data quality, access controls, and compliance metrics. This job description is intended to describe the general nature and level of work performed by an employee assigned to this position. The description is not an exhaustive list of all duties, responsibilities, knowledge, skills, and abilities, and working conditions associated with this position. All requirements are subject to possible modification and reasonably accommodate individuals with disabilities. We offer a comprehensive benefit package. For a complete overview of our benefits package, please visit our Joint Commission Career Page

Requirements

Bachelor’s in computer science, Information Systems, Data Engineering, or related fields.
7–10 years of experience in data engineering, pipeline development, and cloud data platforms.
3+ years of experience leading data engineering teams or projects.
Demonstrated success in delivering enterprise-scale data solutions.
Advanced proficiency in Azure Data Factory, Databricks, Fabric, and MSSQL.
Strong experience with data lake, lakehouse, and data warehouse architectures.
Solid understanding of ETL/ELT processes, data modeling, and metadata management.
Proficiency in SQL and Python; familiarity with cloud platforms and big data technologies.

Nice To Haves

Microsoft Certified: Azure Data Engineer Associate
Databricks Certified Data Engineer
Certified in Governance, Risk and Compliance (ISC²) or similar
Master’s degree in computer science, Information Systems, Data Engineering, or related fields.

Responsibilities

Lead the design, development, and maintenance of scalable data pipelines using Azure Data Factory, Databricks, Foundry, and Fabric.
Build and optimize data ingestion, transformation, and integration processes for diverse data sources.
Develop and manage data lakes, lakehouses, and data warehouses, primarily leveraging MSSQL and cloud-native platforms.
Implement ETL/ELT best practices, ensuring high performance, reliability, and data quality.
Automate data workflows and monitor pipeline health, troubleshooting issues as needed.
Mentor and guide a team of data engineers, fostering technical growth and collaboration.
Work closely with analytics, BI, and business teams to translate requirements into technical solutions.
Promote a culture of innovation, continuous improvement, and knowledge sharing within the team.
Evaluate and implement new data engineering tools and technologies to improve efficiency and scalability.
Establish and enforce standards for code quality, testing, and documentation in data engineering projects.
Champion data quality, reliability, and governance within engineering workflows.
Enable self-service analytics by building reusable data models and accessible data sets.
Collaborate with BI teams to ensure data pipelines meet reporting and dashboard requirements.
Translate business requirements into scalable technical solutions.
Ensure data engineering solutions comply with privacy regulations (e.g., GDPR, HIPAA) and internal security standards.
Monitor and enforce data quality, access controls, and compliance metrics.