About The Position

The Data Platform Manager will design, build, and optimize enterprise-wide data platforms within the Data Warehouse. The role leads and mentors data engineers and works with business stakeholders, data scientists, analysts, and engineering teams to translate requirements into technical specifications and deliver business-critical insights. It is responsible for designing and deploying scalable, high-performance data pipelines in Azure Databricks, maintaining data integrity and availability, and optimizing pipeline performance, cost, and scalability in Azure. The position defines and implements Lakehouse architecture using Delta Lake and Bronze, Silver, and Gold medallion layers. It oversees batch and streaming ETL/ELT pipelines and data transformations, integrates real-time and batch sources, and establishes practices for data ingestion, processing, storage, governance, and data quality. The role also supports production troubleshooting and incident management, operationalizes machine-learning models, implements Unity Catalog security and governance, tunes Spark jobs and Databricks clusters, and integrates Databricks with Azure services and CI/CD workflows.

Requirements

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • 5–7+ years of hands-on data engineering or architecture experience, including at least 2–4 years specifically focused on Azure Databricks and Azure cloud technologies.
  • 2–5 years of experience managing a team of data engineers, data scientists, and/or analysts.
  • Proficiency with relational SQL and NoSQL databases, including document, key-value, graph, and columnar databases.
  • Experience developing and maintaining data models and schemas for data analysis and reporting requirements.
  • Knowledge of distributed-system frameworks such as Apache Hadoop, Spark, or Presto/Trino for processing and retrieving massive data volumes.
  • Understanding of file formats such as Parquet, Avro, and ORC, and compression techniques.
  • Deep proficiency in Python, specifically PySpark, SQL, PowerShell, and Scala.
  • Hands-on experience with Azure Cloud infrastructure, including VNETs, Key Vault, and Identity Management.
  • Deep knowledge of Apache Spark runtime internals, MLflow for MLOps, and orchestration tools such as Airflow.

Nice To Haves

  • Microsoft Certified: Azure Data Engineer Associate (DP-203).
  • Databricks Certified Data Engineer Professional.
  • Azure Solutions Architect Expert.

Responsibilities

  • Lead and mentor a team of data engineers, conduct code reviews, and ensure development standards.
  • Support troubleshooting and incident management for data-related production issues.
  • Collaborate with business stakeholders, data scientists, analysts, and engineering teams to gather requirements, translate them into technical specifications, and deliver business-critical insights.
  • Lead the design, development, and deployment of scalable, high-performance Azure Databricks data pipelines, maintaining data integrity and availability and efficiently extracting, transforming, and loading data from various sources.
  • Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
  • Define best practices for data ingestion, processing, storage, and governance.
  • Implement data quality checks and validation procedures to ensure accuracy and integrity across APIs, databases, and streaming platforms.
  • Collaborate with data scientists and analysts to operationalize and deploy machine-learning models.
  • Define end-to-end Lakehouse architecture using Delta Lake and implement Bronze, Silver, and Gold medallion layers.
  • Oversee robust, scalable, low-latency batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL.
  • Implement data transformations, enrichment, and quality checks using PySpark and Scala in Databricks.
  • Integrate real-time and batch data sources using Apache Kafka and Azure Data Factory.
  • Support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory.
  • Implement Unity Catalog for unified governance, data security, fine-grained RBAC access control, privacy measures, and data lineage tracking.
  • Tune Spark jobs and Databricks clusters to maximize throughput and cost efficiency using auto-scaling and cluster policies.
  • Apply indexing strategies, query optimization, execution plans, and partitioning/sharding.
  • Integrate Databricks with Azure Data Factory, Azure Data Lake Storage Gen2, and Azure DevOps for CI/CD pipelines.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service