Senior Azure Databricks Expert
NewVison
•Remote
About The Position
This role involves designing, building, and optimizing enterprise-grade data platforms on Microsoft Azure. The ideal candidate will bring strong technical expertise in Azure Databricks, Spark, and data engineering, along with the ability to lead teams, mentor junior engineers, and drive best practices across global, cross-functional teams.
Requirements
- Minimum 3+ years of hands-on experience with Azure Databricks
- Strong expertise in Apache Spark, PySpark, and SQL
- Experience in building and optimizing ETL/ELT pipelines
- Hands-on experience with ADLS Gen2 and Delta Lake
- Strong understanding of Spark architecture and performance tuning
- Experience in migration from SQL Server or other relational databases
- Practical experience working in Azure cloud environments
- Strong problem-solving and troubleshooting skills
- Strong communication skills with B2+ English proficiency
- Ability to work in globally distributed teams
- Leadership and mentoring capabilities
- Strong collaboration and stakeholder management skills
- Analytical mindset with attention to detail
Nice To Haves
- Experience with Unity Catalog and data governance frameworks
- Exposure to Azure services such as Azure Key Vault, Azure Monitor, and Networking
- Experience with CI/CD tools and DevOps practices
- Knowledge of Infrastructure as Code (ARM, Terraform, etc.)
- Experience working on enterprise-scale data platforms or analytics projects
- Experience in enterprise data engineering and analytics environments
- Understanding of large-scale data platform modernization initiatives
- Exposure to consulting or client-facing environments is a plus
- Familiarity with Agile/Kanban delivery models (e.g., ServiceNow-based workflows)
Responsibilities
- Provide technical leadership for Azure Databricks-based data transformation initiatives
- Design, develop, and maintain scalable data pipelines using Azure Databricks
- Build and optimize ETL/ELT solutions for enterprise data platforms
- Lead migration and modernization efforts from SQL Server and legacy systems to Azure
- Develop and optimize data processing workflows using Apache Spark, PySpark, and SQL
- Improve performance of Spark workloads through deep understanding of Spark architecture (drivers, executors, partitions, stages, tasks)
- Work with Azure Data Lake Storage Gen2 (ADLS Gen2) and Delta Lake for reliable data storage solutions
- Optimize Databricks clusters, jobs, and workloads for performance and cost efficiency
- Implement data governance and secure access using Unity Catalog and Azure security best practices
- Collaborate with architects, DevOps teams, and business stakeholders to deliver scalable solutions
- Troubleshoot and resolve production issues related to data pipelines and platform performance
- Support CI/CD pipelines and Infrastructure as Code (IaC) implementations
- Mentor junior engineers and provide technical guidance and best practices
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
