Lead Data Engineer - DataBricks
iLink Digital•Milpitas, CA
14h
About The Position
We are seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud -based lakehouse platforms. The role combines hands -on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.
Requirements
- Expert -level Databricks experience (Azure or AWS)
- Strong Spark / PySpark / Spark SQL expertise
- Delta Lake and Lakehouse architecture
- Streaming (Structured Streaming) experience
- Strong experience with Azure or AWS cloud platforms
- Data orchestration tools (ADF, Airflow, or similar)
- Strong SQL and data modeling skills
- Git -based version control
- CI/CD pipelines for data engineering workloads
- Terraform or similar IaC tools
Nice To Haves
- Experience with MLflow and MLOps workflows
- Exposure to Microsoft Fabric or Snowflake
- Databricks certifications (Professional Data Engineer / Architect)
- Experience working in Agile environments
Responsibilities
- Lead the design and implementation of Databricks Lakehouse architectures
- Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
- Drive architectural decisions for batch and streaming data pipelines
- Establish coding standards, best practices, and reusable frameworks
- Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
- Optimize Spark jobs for performance, reliability, and cost
- Implement Delta Lake features (ACID, time travel, schema enforcement)
- Develop and manage Databricks workflows, jobs, and clusters
- Architect Databricks solutions on Azure (preferred) or AWS
- Integrate Databricks with cloud storage and data services
- Enable BI and analytics consumption (Power BI, Tableau)
- Implement data governance using Unity Catalog
- Define RBAC, data access controls, and security best practices
- Enable CI/CD for Databricks using GitHub / Azure DevOps
- Use Infrastructure -as -Code (Terraform) for environment management
- Lead, mentor, and grow data engineering teams
- Conduct design and code reviews
- Collaborate with Data Architects, Product Owners, and stakeholders
- Support production releases, monitoring, and incident resolution
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
