Manager-Data Platforms

Buchanan Ingersoll & Rooney PC•Pittsburgh, PA

21h•Hybrid

About The Position

Buchanan Ingersoll & Rooney is seeking a senior technical leader to design, build, and optimize scalable enterprise data platforms using Databricks Data Warehouse on the firm’s Azure Cloud platform. The position combines deep Databricks expertise with broad knowledge of the Microsoft Azure ecosystem to deliver high-performance data engineering, analytics, and data science solutions. The role calls for hands-on experience with Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, Azure Databricks, and Azure Synapse Data Warehouse. The manager will guide teams and deliver robust data solutions that support actionable insights and decision-making. The position is based at the firm’s corporate Pittsburgh, Pennsylvania location and follows a hybrid schedule of three days in the office and two days remotely from home.

Requirements

5–7+ years of hands-on data engineering or architecture experience, including at least 2–4 years focused specifically on Azure Databricks and Azure cloud technologies.
Bachelor’s degree in Computer Science, Engineering, or a related field.
Proficiency with relational SQL and NoSQL databases, including document, key-value, graph, and columnar databases.
Ability to develop and maintain data models and schemas for data analysis and reporting.
Knowledge of distributed systems and frameworks such as Apache Hadoop, Spark, or Presto/Trino for processing and retrieving large datasets.
Understanding of file formats such as Parquet, Avro, and ORC, and of compression techniques.
Deep proficiency in Python, specifically PySpark, SQL, PowerShell, and Scala.
Hands-on experience with Azure Cloud infrastructure, including VNET networking, Key Vault, and identity management.
Familiarity with data modeling and schema design principles.
Deep knowledge of Apache Spark runtime internals, MLflow for MLOps, and orchestration tools such as Airflow.

Nice To Haves

2–5 years of experience managing a team of data engineers, data scientists, and/or analysts.
Microsoft Certified: Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, or Azure Solutions Architect Expert certification.

Responsibilities

Lead and mentor data engineers, conduct code reviews, and ensure development standards.
Support troubleshooting and incident management for data-related production issues.
Collaborate with business stakeholders, data scientists, and team members to gather requirements and translate them into technical specifications.
Lead the design, development, and deployment of scalable, high-performance Azure Databricks data pipelines, ensuring data integrity, availability, and efficient extraction, transformation, and loading.
Collaborate with data scientists, analysts, and engineering teams to deliver business-critical insights.
Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
Define best practices for data ingestion, processing, storage, and governance.
Implement data quality checks and validation procedures for data from APIs, databases, and streaming platforms.
Collaborate with data scientists and analysts to operationalize and deploy machine learning models.
Define end-to-end Lakehouse architecture using Delta Lake and implement Bronze, Silver, and Gold medallion layers.
Oversee robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL, with minimal latency.
Implement data transformations, enrichment, and quality checks using PySpark or Scala in Databricks.
Integrate real-time and batch data sources using Apache Kafka and Azure Data Factory.
Support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory.
Implement Unity Catalog for unified governance, data security, fine-grained RBAC access control, privacy measures, and data lineage tracking.
Tune Spark jobs and Databricks clusters for throughput and cost efficiency using auto-scaling and cluster policies.
Apply indexing strategies, query optimization, execution plans, and partitioning or sharding.
Integrate Databricks with Azure services, including Azure Data Factory, Azure Data Lake Storage Gen2, and Azure DevOps for CI/CD pipelines.