Lead Databricks Data Engineer

EXL•New York, NY

1d•$100,000 - $135,000

About The Position

We are seeking a Lead Databricks Data Engineer to lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. The role combines hands-on data engineering and solution architecture with technical leadership, stakeholder management, and delivery planning. The engineer will define data engineering standards, mentor teams, drive modernization initiatives, and deliver high-performance cloud-native data solutions supporting analytics, AI/ML, and business intelligence. The position covers Databricks and big data engineering, cloud platform engineering across Azure, AWS, or GCP, DevOps automation, data warehousing, governance, security, and data quality. The posting lists 8–10 years of experience in the summary, 12–15 years of overall IT experience and specific technical experience in the required experience section, and a separate qualifications section listing 5–8 years.

Requirements

The job summary states 8–10 years of experience in data engineering, big data, and cloud data platforms.
The required experience section states 12–15 years of overall IT experience.
At least 6 years of hands-on Databricks experience.
At least 7 years of Spark/PySpark-based data engineering experience.
Experience designing enterprise-scale Data Lake and Lakehouse solutions.
Strong experience leading large data transformation and modernization programs.
Experience processing structured, semi-structured, and unstructured data.
Experience supporting analytics, AI/ML, and reporting ecosystems.
Proven experience managing technical teams and driving architectural decisions.
Strong experience in one or more cloud platforms: Azure, AWS, or GCP.
Knowledge of Databricks Lakehouse Platform, Databricks Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables, and Databricks Asset Bundles.
Knowledge of Apache Spark, PySpark, Spark SQL, Kafka, real-time streaming architectures, and data lake/lakehouse architectures.
Programming skills in Python, SQL, PySpark, and shell scripting.
Experience with DevOps and automation tools such as Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes.
Knowledge of dimensional modeling, Star and Snowflake schemas, Data Vault, Master Data Management, and data governance frameworks.
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field is listed as a preferred qualification; the final qualifications section also lists a Bachelor's/Master's in Engineering.
The final qualifications section lists 5–8 years, conflicting with the experience figures elsewhere in the posting.

Nice To Haves

Scala.
Databricks Certified Data Engineer Professional certification.
Azure Data Engineer Associate certification.
AWS Data Analytics Specialty certification.
Google Professional Data Engineer certification.
TOGAF or cloud architecture certifications.
Exposure to MLOps and AI Engineering.
Snowflake and Databricks integration experience.
Microsoft Fabric exposure.
Data Mesh implementation experience.
Experience with data governance tools such as Collibra, Alation, or Informatica.
Expertise in data quality frameworks such as Great Expectations or Deequ.

Responsibilities

Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
Architect and develop scalable ETL/ELT pipelines for batch and real-time processing.
Define enterprise data models, ingestion frameworks, and transformation strategies.
Lead implementation of Delta Lake, Unity Catalog, and data governance frameworks.
Build and optimize scalable Spark and PySpark solutions for large-volume datasets.
Drive cloud data platform modernization and migration initiatives.
Serve as technical lead for Databricks and data engineering programs.
Establish engineering best practices, coding standards, and architecture guidelines.
Conduct architecture reviews, code reviews, and solution assessments.
Mentor and guide Data Engineers, Technical Leads, and junior team members.
Collaborate with Enterprise Architects, Product Owners, Data Scientists, and business stakeholders.
Design cloud-native data solutions on Azure, AWS, or GCP.
Implement secure, scalable, and resilient data architectures.
Optimize data platform cost, performance, reliability, and availability.
Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
Define and implement monitoring, observability, and operational excellence practices.
Establish data governance, lineage, cataloging, and security controls.
Ensure compliance with organizational and regulatory requirements.
Implement data quality frameworks and monitoring solutions.
Work with security and compliance teams to enforce data protection standards.
Partner with business leaders to understand strategic data requirements and translate them into technical solutions.
Drive delivery planning, risk management, and resource coordination.
Present architecture decisions and technical recommendations to leadership teams.
Design data pipelines, optimize data processing and storage, ensure performance standards, provide technical support, and collaborate with stakeholders.