Databricks Engineer

iLink Digital•Milpitas, CA

About The Position

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Requirements

Databricks Lakehouse Platform
Apache Spark
PySpark
Spark SQL
Delta Lake
Python
SQL
Microsoft Azure (preferred)
AWS
Google Cloud Platform
Azure Data Factory (ADF)
Azure Data Lake Storage (ADLS Gen2)
Azure Synapse Analytics
Azure Key Vault
Azure DevOps
Data Engineering
Data Warehousing
Data Modeling
ETL/ELT Development
Batch Processing
Streaming (Kafka/Event Hubs)
Data Lake Architecture
Git
Azure DevOps / GitHub
CI/CD Pipelines

Nice To Haves

Experience with Unity Catalog.
Knowledge of Databricks Workflows and Jobs.
Hands-on experience with Delta Live Tables (DLT).
Exposure to MLflow is an added advantage.
Experience with data governance and security best practices.
Familiarity with Infrastructure as Code (Terraform) is a plus.
Experience with real-time analytics.
Knowledge of Lakehouse architecture.
Experience with Agile/Scrum methodologies.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management abilities.

Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks.
Build ETL/ELT workflows for batch and streaming data processing.
Develop solutions using PySpark, Spark SQL, and Delta Lake.
Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
Implement CI/CD pipelines and deployment automation for Databricks workloads.
Ensure data quality, security, governance, and compliance.
Monitor, troubleshoot, and optimize production data pipelines.
Document technical solutions and follow engineering best practices.