Lead Databricks Data Engineer

EXL•New York, NY

1d•$130,000 - $160,000

About The Position

We are seeking a Lead Databricks Data Engineer to lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. The role combines hands-on data engineering with solution architecture, technical leadership, stakeholder management, and delivery responsibilities. The successful candidate will define data engineering standards, mentor teams, drive modernization initiatives, and deliver high-performance cloud-native data solutions supporting analytics, AI/ML, and business intelligence. The position covers architecture and implementation of scalable batch and real-time ETL/ELT pipelines; data models, ingestion frameworks, and transformation strategies; and platforms using Databricks, Delta Lake, Unity Catalog, Spark, and cloud services. It also leads cloud modernization and migration, CI/CD and infrastructure-as-code implementation, platform monitoring and operational practices, data governance and quality, and security and compliance efforts. The role partners with business and technical stakeholders to translate strategic data needs into solutions, coordinate delivery and risks, and present architecture recommendations to leadership.

Requirements

The job description lists 10–12 years of experience in the job summary and 12–15 years of overall IT experience in the required experience section; it also lists 15–18 years in the final qualifications section.
Minimum 6 years of hands-on Databricks experience.
At least 7 years of experience in Spark/PySpark-based data engineering.
Experience designing enterprise-scale Data Lake and Lakehouse solutions.
Strong experience leading large data transformation and modernization programs.
Experience processing structured, semi-structured, and unstructured data.
Experience supporting analytics, AI/ML, and reporting ecosystems.
Proven experience managing technical teams and driving architectural decisions.
Strong experience with one or more cloud platforms: Azure, AWS, or GCP.
Required technical skills include Databricks Lakehouse Platform, Databricks Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables, and Databricks Asset Bundles.
Required big data skills include Apache Spark, PySpark, Spark SQL, Kafka, real-time streaming architectures, and data lake and lakehouse architectures.
Programming skills include Python, SQL, PySpark, and Shell Scripting.
DevOps and automation skills include Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes.
Data warehousing knowledge includes dimensional modeling, star and snowflake schemas, Data Vault, master data management, and data governance frameworks.
A graduate degree with a four-year BS or a professional degree is listed in the qualifications section.

Nice To Haves

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.
Databricks Certified Data Engineer Professional certification.
Azure Data Engineer Associate certification.
AWS Data Analytics Specialty certification.
Google Professional Data Engineer certification.
TOGAF or cloud architecture certifications.
MLOps and AI Engineering exposure.
Snowflake and Databricks integration experience.
Microsoft Fabric exposure.
Data Mesh implementation experience.
Experience with data governance tools such as Collibra, Alation, or Informatica.
Expertise in data quality frameworks such as Great Expectations or Deequ.
Scala.

Responsibilities

Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
Architect and develop scalable ETL/ELT pipelines for batch and real-time processing.
Define enterprise data models, ingestion frameworks, and transformation strategies.
Lead implementation of Delta Lake, Unity Catalog, and data governance frameworks.
Build and optimize scalable Spark and PySpark solutions for large-volume datasets.
Drive cloud data platform modernization and migration initiatives.
Serve as technical lead for Databricks and data engineering programs.
Establish engineering best practices, coding standards, and architecture guidelines.
Conduct architecture reviews, code reviews, and solution assessments.
Mentor and guide Data Engineers, Technical Leads, and junior team members.
Collaborate with Enterprise Architects, Product Owners, Data Scientists, and business stakeholders.
Design cloud-native data solutions on Azure, AWS, or GCP.
Implement secure, scalable, and resilient data architectures.
Optimize data platform costs, performance, reliability, and availability.
Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
Define and implement monitoring, observability, and operational excellence practices.
Establish data governance, lineage, cataloging, and security controls.
Ensure compliance with organizational and regulatory requirements.
Implement data quality frameworks and monitoring solutions.
Collaborate with security and compliance teams to enforce data protection standards.
Partner with business leaders to understand strategic data requirements and translate them into scalable technical solutions.
Drive delivery planning, risk management, and resource coordination.
Present architecture decisions and technical recommendations to leadership teams.
Manage database systems and address performance tuning, security, and compliance standards.