Azure Databricks Engineer

Opinov8 • St. Louis, MO

2d • Remote

About The Position

We are looking for a skilled Azure Databricks Engineer to design, develop, and optimize scalable data pipelines and cloud-based analytics solutions using Microsoft Azure and Databricks technologies. The ideal candidate should have strong experience in big data processing, ETL/ELT workflows, PySpark, and Azure cloud services.

Requirements

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field
• 5+ years of experience in Data Engineering or Big Data development
• Strong hands-on experience with Azure Databricks and Apache Spark
• Proficiency in PySpark, Spark SQL, and SQL
• Experience with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS)
• Knowledge of Delta Lake, Lakehouse architecture, and data modeling concepts
• Experience with distributed data processing and performance tuning
• Familiarity with CI/CD pipelines, Git, and Azure DevOps

Nice To Haves

• Experience with Kafka, dbt, Airflow, or Snowflake.
• Knowledge of Terraform or Infrastructure as Code (IaC).
• Exposure to MLflow, machine learning pipelines, or MLOps.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Databricks and Apache Spark
• Develop and optimize PySpark/Spark SQL notebooks for large-scale data processing
• Implement batch and real-time data ingestion pipelines using Azure Data Factory (ADF), Databricks Workflows, and Structured Streaming
• Manage and optimize Delta Lake/Lakehouse architecture, including Bronze, Silver, and Gold layers
• Integrate data from multiple sources including APIs, databases, cloud storage, and third-party systems
• Work with Azure services such as ADLS Gen2, Azure Synapse Analytics, Azure SQL Database, and Azure DevOps
• Monitor, troubleshoot, and improve pipeline performance, scalability, and cost optimization
• Implement data governance, security, access controls, and compliance standards using Unity Catalog and RBAC
• Collaborate with Data Engineers, Data Scientists, Analysts, and Product teams to deliver reliable data solutions
• Support CI/CD implementation, automation, and infrastructure-as-code practices

Benefits

• Digital-First Approach: Great talent knows no borders! You can work from wherever you are — we hire and collaborate with professionals worldwide.
• Remote Work Model: Balance your professional and personal life with our flexible working conditions, empowering you to deliver your best from anywhere.
• Exciting Projects: Dive into impactful projects across industries that challenge and spark creativity.
• Boost Your Expertise: Grow your career with continuous learning, development opportunities, and hands-on experience.
• Join the Best Team Ever: Collaborate with our diverse and cross-cultural team of passionate technologists and creative thinkers.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior