Lead Data Engineer, Databricks (Remote, US)
About The Position
Lead Data Engineers are the technical backbone of V4C delivery. They lead the engineering workstream on customer engagements, translating architecture into production pipelines, establishing team patterns, and owning the quality of delivered work. The role spans batch and streaming workloads and complex enterprise source systems, with real delivery deadlines and direct customer communication about the implementation. Compared with a Data Engineer, this role carries ownership of a workstream and accountability for the engineers on it. The position works directly with customers and often with Databricks field teams.
Requirements
- At least 7 years of data engineering experience, plus experience leading projects or teams.
- Strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
- Production experience building both batch and streaming pipelines at enterprise scale.
- Hands-on experience with AWS, Azure, or GCP and its data services.
- Experience with data modeling, Git, and CI/CD for data workloads.
- Demonstrated experience optimizing Spark performance and managing platform cost.
- Strong client communication skills, including the ability to explain technical trade-offs to non-technical stakeholders.
Nice To Haves
- Healthcare or life sciences experience.
- Existing Databricks Certified Data Engineer Associate or Professional certification.
- Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles.
- Experience with dbt, Airflow, or equivalent orchestration and transformation tooling.
- Experience migrating a legacy ETL estate to a modern cloud platform.
- Consulting or client-facing engineering background.
Responsibilities
- Lead engineering on customer engagements from design through production handover.
- Build and optimize batch and streaming pipelines using PySpark, SQL, and Delta Lake on Databricks.
- Implement Medallion architecture across cloud ecosystems and establish the layer conventions followed by the team.
- Own performance and cost by benchmarking Spark jobs, tuning SQL, right-sizing compute, and explaining trade-offs to customers.
- Implement data quality and validation as code, including Lakeflow expectations and automated CI testing.
- Mentor mid-level and junior engineers through code review, pairing, and design feedback.
- Partner with the Data Architect on design decisions and escalate when implementation reveals that the design will not hold.
Benefits
- Unlimited PTO.
- Medical, dental, and vision health care plan.
- Retirement plan (401k, IRA).
- Life insurance (basic, voluntary, and AD&D).
- Family leave (maternity and paternity).
- Short-term and long-term disability.
- Training and development.
- A personalized certification roadmap taught in-house by Databricks-approved trainers.
- A dedicated Employee Success Manager focused on the employee experience.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
