Lead Data Engineer

Emory•Atlanta, GA

1d•Remote

About The Position

The Lead Data Engineer provides technical leadership and hands-on engineering expertise for developing and operating Emory University’s Unified Data Platform (UDP), built on Microsoft Fabric. The role integrates data from HR, Finance, Student, Research, and other enterprise applications to support analytics, reporting, research, and institutional decision-making. Working with architects, researchers, business stakeholders, source system owners, and cross-functional teams, the Lead Data Engineer translates requirements into reliable, scalable, governed data solutions. The position guides engineering delivery, mentors team members, establishes standards, resolves complex technical issues, and contributes directly to pipeline development and production support. Microsoft Fabric experience is preferred. Candidates with strong Azure Synapse or Databricks experience using PySpark, and the ability to apply that experience to Microsoft Fabric, will also be considered. The role requires sound judgment in data quality, security, governance, and the handling of sensitive university and research data. Tasks can be performed remotely with only occasional visits to an Emory University location. Eastern time zone business hours may apply. Emory does not approve New Jersey, Alaska, Hawaii, U.S. Territories, or locations outside the United States as primary work locations.

Requirements

A bachelor’s degree in a related field and seven years of related experience, or an equivalent combination of education, training, and experience.
Strong experience designing and maintaining data pipelines and ETL/ELT processes across multiple source systems.
Proficiency in Python, SQL, and Apache Spark (PySpark).
Experience with modern data platforms such as Azure Fabric, Azure Synapse Serverless, or Databricks.
Ability to work with large, complex, and distributed datasets.
Solid understanding of data modeling concepts and best practices.
Familiarity with data governance, metadata management, and data quality frameworks.
Ability to lead technical design discussions, review code, mentor engineers, and guide delivery across multiple workstreams.
Experience with CI/CD practices and proficiency with version control systems such as Git.
Familiarity with data pipeline orchestration and monitoring tools such as Airflow, Azure Data Factory, or similar frameworks.
Strong analytical, problem-solving, and critical-thinking abilities.
Strong communication and collaboration skills, including the ability to explain technical concepts to diverse audiences.
Ability to prioritize engineering work, identify delivery risks, and adapt to changing business and platform requirements.
Ability to manage workload and report task status in a timely manner.
Ability to guide stakeholders on solutions and support structures and to support prioritization governance.
Ability to follow standard operational procedures and HIPAA regulations.

Nice To Haves

Microsoft Fabric experience, including Fabric Lakehouse, Warehouse, notebooks, and data pipelines.
Experience integrating higher education and healthcare enterprise HR, Finance, Student, and Research applications.
Understanding of biomedical informatics methodologies.

Responsibilities

Lead technical delivery of source system onboarding and data integration initiatives, from source analysis and design through development, validation, deployment, and production support.
Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards aligned with UDP architecture and institutional priorities.
Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, and Microsoft Fabric or comparable cloud technologies.
Analyze source schemas, business rules, and dependencies across enterprise applications; define source-to-target mappings, incremental load strategies, and reconciliation requirements.
Guide implementation of bronze, silver, and gold data layers, including data transformation, dimensional modeling, and curated datasets for reporting and analytics.
Establish and review standards for coding, testing, documentation, version control, and deployment; conduct code reviews and mentor data engineers.
Coordinate engineering tasks and technical dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source system owners; communicate delivery progress, risks, and resource needs.
Implement data quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures for reliable production processing.
Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration to improve performance, scalability, and efficient use of platform capacity.
Apply institutional data governance and security standards, including access controls, metadata, lineage, and appropriate handling of sensitive data; support HIPAA requirements where applicable.
Work with researchers and business stakeholders to translate requirements into technical specifications and explain design choices and data limitations.
Create and maintain technical documentation, operational runbooks, source mappings, and support procedures; coordinate production deployments and knowledge transfer.
Support existing UDP and Enterprise Business Intelligence (EBI) integrations and services, balancing operational needs with project delivery priorities.
Evaluate emerging technologies and develop focused proofs of concept addressing identified business or engineering needs.
Contribute as a positive team member to project delivery, helping deliver quality applications and components within scope, on time, within budget, and in compliance.
Manage workload effectively and report task status in a timely manner.
Guide stakeholders on solutions and support structures; ensure appropriate governance when prioritizing partner work.
Form tactical strategies to support multiple clients and lead small- to medium-sized technical teams.
Mentor other analysts on regulatory adherence and follow standard operating procedures and HIPAA regulations.
Develop strategies for managing complex datasets by maintaining data standards and metadata.
Apply biomedical informatics technical standards, methodologies, and principles to research-specific program needs, objectives, and outcomes.
Develop reporting infrastructure to meet multiple client needs and generate standard templates for architecture documentation related to current and proposed informatic solutions.
Perform other related duties as required.