Lead Data Engineer

Humana

1d•Remote

About The Position

Humana is seeking an experienced and strategic Lead Data Engineer to join their ITSM Applied AI and Analytics team. This individual will lead the development and evolution of an enterprise-facing data store built on Azure Databricks, driving architectural decisions, mentoring team members, and ensuring scalable, high-quality data solutions across the organization. The position reports to an Associate Director, Data Engineering.

Requirements

5+ years of experience in data engineering, with at least 2 years in a lead or senior role.
Deep expertise in Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Strong understanding of Medallion Architecture and scalable data design.
Strong proficiency in Python for data engineering tasks (APIs, Spark data frames, OAuth, etc.).
Familiarity with Databricks Materialized views.
Experience with Kafka for real-time data ingestion and processing.
Proficiency in GitHub, CI/CD pipelines, and DevOps practices.
Experience building and integrating REST APIs.
Excellent communication, leadership, and documentation skills.

Nice To Haves

Databricks certifications (e.g., Data Engineer Professional).
Familiarity with ServiceNow data structures and integration.
Experience with enterprise analytics platforms like Power BI or Power Apps.
Background in enterprise environments with data governance and compliance requirements.
Background in enterprise data architecture and governance.

Responsibilities

Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Apache Spark, Unity Catalog, and Delta Lake.
Architect data workflows following Medallion Architecture (Bronze, Silver, Gold layers).
Guide the integration of external APIs and systems, including ServiceNow and Kafka for real-time data streaming.
Establish and enforce best practices in CI/CD, GitHub workflows, and code quality.
Manage compute appropriately to ensure cost control.
Collaborate with stakeholders to define data strategy, governance, and platform enhancements.
Provide technical leadership and mentorship to data engineers and analysts.
Conduct code reviews and ensure adherence to engineering standards.
Communicate effectively with technical and non-technical stakeholders.