Azure Databrick SME

BayRock Labs • Milpitas, CA

4h

About The Position

We are seeking a highly skilled Data Platform Engineer (SME level) with deep hands-on expertise in Azure Databricks, data engineering, and platform engineering. The ideal candidate will have strong experience in Microsoft Azure, DevOps, and automation, along with working knowledge of AI/ML Ops platforms. This role is critical in designing, building, and maintaining scalable, secure, and high-performance enterprise data solutions.

Requirements

• Expert-level hands-on experience with Azure Databricks (clusters, notebooks, Delta Lake, MLflow).
• Strong background in data engineering (ETL, data modeling, big data frameworks).
• Proven expertise in platform engineering and enterprise-scale data solutions.
• Proficiency in Microsoft Azure services (Data Lake, Synapse, Key Vault, Functions, Logic Apps).
• Solid understanding of DevOps practices (CI/CD pipelines, Git, Terraform/ARM templates, automation).
• Working knowledge of AI/ML Ops platforms (model deployment, monitoring, ML lifecycle management).
• Strong skills in SQL, Python, and Spark for data processing and analytics.
• Excellent problem-solving, communication, and stakeholder management skills.
• 10+ years of experience in enterprise data engineering/platform roles preferred.

Nice To Haves

• Experience with enterprise supply chain or manufacturing data platforms.
• Familiarity with data governance frameworks and compliance standards.
• Certifications in Azure Data Engineer Associate or Azure Solutions Architect Expert.

Responsibilities

• Build and maintain enterprise-grade data platforms leveraging Azure Databricks.
• Design, develop, and optimize data pipelines, ETL workflows, and orchestration processes.
• Support platform engineering initiatives to enhance scalability, reliability, and performance.
• Integrate AI/ML Ops capabilities into data platforms to enable advanced analytics and machine learning.
• Implement DevOps and automation best practices for CI/CD, monitoring, and infrastructure as code.
• Ensure security, compliance, and governance across all data systems.
• Collaborate with cross-functional teams (data scientists, architects, business stakeholders) to deliver end-to-end solutions.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed