Senior Azure Databricks Engineer

VSG Business Solutions • Jersey City, NJ

2d • Onsite

About The Position

Seeking experienced Databricks/Azure consultants to support the design, implementation, and ongoing development of the Databricks data platform. The consultants will work with technology and data teams to build and enhance scalable data solutions using Databricks and Microsoft Azure, with a strong focus on data engineering, integration, security, automation, and platform best practices.

Candidates should have hands-on experience across the Databricks ecosystem, including Databricks Workspace, Unity Catalog, Delta Lake, Spark, and MLflow, as well as experience developing solutions in Microsoft Azure. They should be comfortable working in a large enterprise environment and collaborating across application development, infrastructure, cloud, security, architecture, and data teams.

The role is onsite in Jersey City, New Jersey, at least four days per week initially; the employer may consider three days onsite later. The interview process has two rounds, one of which is in person.

Requirements

• Hands-on experience with Databricks Workspace, Unity Catalog, Delta Lake, Spark UI, and MLflow.
• Strong development experience with Python, PySpark, SQL, and/or Java.
• Strong Microsoft Azure experience, particularly Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Data Factory.
• Experience developing, troubleshooting, and performance-tuning Spark workloads and data processing pipelines.
• Experience with Git, CI/CD pipelines, OpenShift, Kubernetes, and Terraform or Infrastructure as Code (IaC).
• Experience implementing or supporting Databricks security, access controls, secrets management, identity integration, data governance, and Unity Catalog.
• Experience with Databricks workflows, job orchestration, monitoring, troubleshooting, performance optimization, and production support.
• Ability to collaborate across technical teams and work within established enterprise architecture, security, change-management, and governance standards.

Nice To Haves

• Experience with AWS, including S3 and IAM.
• Knowledge of Apache Airflow.
• Experience designing and delivering production-grade Databricks solutions in a large enterprise environment, particularly where security, data governance, resiliency, automation, and operational support are key requirements.

Responsibilities

• Design, implement, and support the ongoing development of the Databricks data platform.
• Build and enhance scalable data solutions using Databricks and Microsoft Azure.
• Develop and optimize data processing and ETL/ELT pipelines.
• Develop, troubleshoot, and performance-tune Spark-based workloads and data processing pipelines.
• Support Databricks security, access controls, secrets management, identity integration, data governance, and Unity Catalog.
• Automate and operate Databricks workflows and job orchestration; monitor, troubleshoot, optimize performance, and provide production support.
• Collaborate across application development, infrastructure, cloud, security, architecture, and data teams.
• Deliver within enterprise architecture, security, change-management, and governance standards.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed