Senior Data Platform Engineer - Azure Databricks
About The Position
We are seeking a skilled Senior Data Platform Engineer with hands-on experience in Azure Databricks and a strong background in healthcare environment. The role is responsible for designing, building, and optimizing our enterprise data platform. The ideal candidate will be responsible for maintaining scalable data processing environments in Azure, while ensuring seamless integration across Azure Databricks, DevOps, HVR/ Fivetran, Snowflake, Collibra, Power BI, and Tableau ecosystems. The ideal candidate will have deep experience in cloud engineering at least in one of the major cloud providers (Azure, AWS or GCP), strong hand-on experience of Kubernetes, containers, serverless and modern architectures, experience with Infrastructure as code (Terraform/ Ansible), solid understanding of networking security, IAM, monitoring and automation. This role bridges data architecture, data engineering, and data governance — enabling secure, scalable, efficient and resilient cloud data platform foundation.
Requirements
- Bachelor’s or Master’s degree in computer science, Information Technology, Data Engineering, or related field
- At least four to seven (4 – 7) years of hands-on experience with Azure Databricks in enterprise environments
- Strong proficiency in PySpark, SQL, and data lakehouse architecture
- Deep understanding of Azure ecosystem – including Azure Data Lake, ARO, AKS, Key Vault, Azure Ansible and Azure DevOps
- Experience with CI/CD implementation using Azure DevOps
- Working knowledge of pyspark, Collibra, Power BI, and Tableau integration workflows
- Familiarity with data governance, security, and compliance best practices
- Proficiency in scripting languages (Python, Bash, PowerShell)
- Excellent problem-solving and cross-functional collaboration skills
- Experience with Terraform or other IaC tools for Azure
- Familiarity with Delta Live Tables, Unity Catalog and MLflow
Nice To Haves
- Familiarity with DQ frameworks, or observability tools in Azure/Databricks
- Knowledge of data governance, metadata management, and audit controls in a healthcare setting
- Experience with Lakehouse architecture and medallion design patterns
- Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert
- Exposure to Agile/Scrum delivery models
Responsibilities
- Design, deploy, and maintain scalable Databricks clusters and workspaces within Azure
- Automate cluster management, job orchestration, and environment provisioning using Infrastructure-as-Code (IaC) tools (e.g., Terraform, ARM, or Bicep)
- Optimize Spark jobs, delta pipelines, and data lakehouse performance
- Implement robust CI/CD pipelines for Databricks workflows and notebooks
- Manage environment configurations, libraries, secrets, and access controls
- Implement and manage Azure DevOps pipelines for Databricks code deployment, testing, and release management
- Develop reusable CI/CD templates for data ingestion and transformation workflows
- Collaborate with development teams to enable version control, code quality checks, and automated testing
- Support and integrate Databricks and/or dbt (Data Build Tool) workflows within Databricks for modular SQL-based transformations
- Ensure efficient orchestration of ETL/ELT pipelines and integration with data lake and data warehouse environments
- Integrate Databricks and Azure data assets with Collibra for data cataloging, lineage tracking, and governance
- Work closely with data governance teams to ensure compliance with data quality and metadata management standards
- Enable secure and performant data access for Power BI and Tableau through optimized data models and APIs
- Collaborate with BI developers to design efficient datasets and dashboards
- Work with stakeholders, including the Product, Data and Design teams to assist with platform-related technical issues and support their data needs
- Strong team collaboration and experience working with remote teams
- Mentor and guide other platform engineers through reviews, training etc
- Support implementing data quality, validation, and governance processes to ensure compliance with healthcare regulations (e.g., HIPAA)
- Understand and deploy best practices in data engineering
Benefits
- Flexible work solutions including remote options, hybrid work schedules
- Competitive pay
- Paid time off including holidays and volunteer events
- Health insurance coverage for you and your dependents on Day 1
- 401(k)
- Tuition reimbursement
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
