Azure Databricks

HIGH BRIDGE CONSULTING LLC•New York, NY

26d•$80 - $90•Hybrid

About The Position

We are seeking an experienced Data Engineer – Azure Databricks (Corporate Vertical Developer) to support multiple corporate business functions, including RESO, HCG, Compliance, Facilities, and Finance. This role will be responsible for designing and building scalable data solutions, enabling enterprise-wide analytics, and driving data integration initiatives across the organization. The ideal candidate will have strong expertise in Azure cloud technologies, Databricks, data engineering, and enterprise data integration, with the ability to collaborate effectively across business and technical teams.

Requirements

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience).
5+ years of experience in Data Engineering or a related discipline.
Strong hands-on experience with Microsoft Azure data services.
Extensive experience with Azure Databricks, including Apache Spark and distributed data processing.
Advanced SQL development and optimization skills with large-scale data environments.
Experience building and orchestrating data pipelines using Azure Data Factory (ADF) or similar ETL/ELT tools.
Strong experience with Azure SQL Database, including performance tuning and optimization.
Proficiency in Python and/or Scala for data engineering and automation.
Experience with C# and PowerShell for integration and automation tasks.
Experience developing and deploying Azure Functions for data processing and system integrations.
Experience integrating APIs, event-driven architectures, and enterprise applications.
Strong understanding of scalable, secure, and cloud-native data architecture principles.
Excellent communication and collaboration skills with the ability to work across multiple business domains.

Nice To Haves

Experience with Azure DevOps and CI/CD pipeline implementation.
Knowledge of Delta Lake and modern Lakehouse architectures.
Familiarity with data governance, metadata management, and data catalog solutions.
Experience working within Agile/Scrum development environments.
Relevant Microsoft Azure and/or Databricks certifications.

Responsibilities

Design, develop, and maintain scalable data pipelines using Microsoft Azure and Azure Databricks.
Build and optimize ETL/ELT processes for both structured and unstructured data sources.
Partner with stakeholders across corporate functions to gather requirements and deliver data-driven solutions.
Design and implement data models, architectures, and frameworks that support reporting, analytics, and business intelligence initiatives.
Integrate enterprise platforms such as ServiceNow, Dataverse, Workday, and other business applications into centralized data environments.
Ensure adherence to data quality, governance, security, and compliance standards.
Monitor, troubleshoot, and optimize data workflows to ensure high availability and performance.
Contribute to the ongoing enhancement of enterprise data platforms and recommend innovative technologies and best practices.
Support data modernization initiatives and enterprise-wide analytics strategies.