Information Technology_USA - USA_Developer

Real Soft•Jacksonville, FL

5d•Onsite

About The Position

This role is for a Data Engineer specializing in Databricks. The candidate must be local to Boston for in-person evaluations and will be responsible for developing and optimizing Databricks solutions. The position requires extensive experience in data engineering, ETL/ELT, SQL, and cloud-based data platforms.

Requirements

At least 8+ years of extensive Data Engineering Experience in Databricks, ETL/ELT using Data pipelines, SQL/Procedure experience.
8+ Years of hands-on development proficiency in implementing Databricks solutions using Scala and Spark with Data frames and Notebooks/SQL.
8+ Years of hands-on development and performance tuning/enhancement experience in Scala/PySpark and SQL.
8+ Years of hands-on development experience in working on Azure Cloud.
Strong experience with Databricks, including developing and optimizing Spark jobs, data transformations, and data processing workflows.
Experience and a good understanding of Databricks Delta Lake storage, Unity Catalog, Autoloader, Lakehouse Architecture, Medallion Architecture, Databricks Serverless Options, Dataset and Data frame, Databricks Clusters Configuration and Sizing, Databricks Job Orchestration, Spark Structured Streaming.
Experience with complex and large volume data streaming and data transformations using Azure Databricks.
Extensive experience in Datawarehouse/Data Lakehouse implementations.
Experience with monitoring, debugging, and resolving issues in live Databricks jobs.
Strong hands-on expertise in troubleshooting DevOps pipelines, Azure, and AKS services.
Must be local to Boston for in-person evaluation.

Nice To Haves

Python

Responsibilities

Provide Data Ingestion, Streaming and Transformations solutions and do development work.
Develop and optimize Spark jobs, data transformations, and data processing workflows using Databricks.
Implement Databricks solutions using Scala and Spark with Data frames and Notebooks/SQL.
Perform development and performance tuning/enhancement in Scala/PySpark and SQL.
Work on Azure Cloud for Databricks implementations.
Develop and optimize solutions using Databricks Delta Lake storage, Unity Catalog, Autoloader, Lakehouse Architecture, Medallion Architecture, and Databricks Serverless Options.
Configure and size Databricks Clusters.
Orchestrate Spark Structured Streaming jobs.
Handle complex and large volume data streaming and data transformations using Azure Databricks.
Implement Datawarehouse/Data Lakehouse solutions.
Monitor, debug, and resolve issues in live Databricks jobs.
Troubleshoot DevOps pipelines, Azure, and AKS services.