Lead Data Engineer

Emory University•Atlanta, GA

17h•Remote

About The Position

The Lead Data Engineer provides technical leadership and hands-on engineering expertise for the development and operation of Emory University’s Unified Data Platform (UDP), built on Microsoft Fabric. The role leads integration of data from HR, Finance, Student, Research, and other enterprise applications to support analytics, reporting, research, and institutional decision-making. Working with architects, researchers, business stakeholders, and cross-functional teams, the lead translates requirements into reliable, scalable, and governed data solutions. The position guides engineering delivery, mentors team members, establishes development standards, resolves complex technical issues, and contributes directly to pipeline development and production support. Microsoft Fabric experience is preferred; candidates with strong Azure Synapse or Databricks experience using PySpark may also be considered. The role requires sound judgment in data quality, security, governance, and handling sensitive university and research data. The position also contributes to reporting infrastructure, biomedical informatics and research-specific data needs, and existing UDP and Enterprise Business Intelligence integrations and services. Tasks may be performed remotely with only occasional visits to an Emory University location. Eastern time zone business hours may apply. Emory does not approve New Jersey, Alaska, Hawaii, U.S. territories, or locations outside the United States as primary work locations.

Requirements

Bachelor’s degree in a related field and seven years of related experience, or an equivalent combination of education, training, and experience.
Strong experience designing and maintaining data pipelines and ETL/ELT processes across multiple source systems.
Proficiency in Python, SQL, and Apache Spark (PySpark).
Experience with modern data platforms such as Azure Fabric, Azure Synapse Serverless, or Databricks.
Ability to work with large, complex, and distributed datasets.
Solid understanding of data-modeling concepts and best practices.
Familiarity with data governance, metadata management, and data-quality frameworks.
Ability to lead technical design discussions, review code, mentor engineers, and guide delivery across multiple workstreams.
Experience with CI/CD practices and proficiency with version-control systems such as Git.
Familiarity with data-pipeline orchestration and monitoring tools such as Airflow, Azure Data Factory, or similar frameworks.
Strong analytical, problem-solving, and critical-thinking abilities.
Strong communication and collaboration skills, including the ability to explain technical concepts to diverse audiences.
Ability to prioritize engineering work, identify delivery risks, and adapt to changing business and platform requirements.
Ability to manage workload effectively and report task status in a timely manner.
Understanding of and ability to follow standard operational procedures and HIPAA regulations.

Nice To Haves

Microsoft Fabric experience, including Lakehouse, Warehouse, notebooks, and data pipelines.
Experience integrating higher education and healthcare enterprise HR, Finance, Student, and Research applications.
Understanding of biomedical informatics methodologies.
Experience applying biomedical informatics technical standards, methodologies, and principles to research-specific program needs.

Responsibilities

Lead source-system onboarding and data integration from analysis and design through development, validation, deployment, and production support.
Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards aligned with UDP architecture and institutional priorities.
Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, Microsoft Fabric, or comparable cloud technologies.
Analyze source schemas, business rules, and enterprise application dependencies; define source-to-target mappings, incremental load strategies, and reconciliation requirements.
Guide implementation of bronze, silver, and gold data layers, including transformations, dimensional modeling, and curated datasets for reporting and analytics.
Establish and review coding, testing, documentation, version-control, and deployment standards; conduct code reviews and mentor data engineers.
Coordinate engineering tasks and technical dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source-system owners; communicate progress, risks, and resource needs.
Implement data-quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures for reliable production processing.
Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration for performance, scalability, and efficient platform-capacity use.
Apply institutional data-governance and security standards, including access controls, metadata, lineage, and appropriate handling of sensitive data; support HIPAA requirements where applicable.
Translate stakeholder requirements into technical specifications and explain design choices and data limitations clearly.
Create and maintain technical documentation, operational runbooks, source mappings, and support procedures; coordinate production deployments and knowledge transfer.
Support existing UDP and Enterprise Business Intelligence integrations and services while balancing operational needs with project delivery priorities.
Evaluate emerging technologies, develop focused proofs of concept, and promote technologies that address identified business or engineering needs.
Work as a positive member of cross-functional projects to deliver quality applications and components within scope, schedule, and budget, and ensure compliance.
Manage workload, report task status in a timely manner, guide stakeholders on solutions and support structures, and help prioritize partner work.
Form tactical strategies to support multiple clients and lead small- to medium-sized technical teams; mentor analysts on regulatory adherence.
Follow standard operating procedures and HIPAA regulations; maintain data standards and metadata for complex datasets.
Apply biomedical informatics technical standards, methodologies, and principles to research program needs, objectives, and outcomes.
Develop reporting infrastructure for multiple client needs and generate standard architecture-documentation templates for current and proposed informatics solutions.
Perform other related duties as required.