Manager-Data Platforms

Buchanan Ingersoll-Rooney • Pittsburgh, PA

1d • Hybrid

About The Position

Buchanan Ingersoll & Rooney is seeking a senior technical leader to design, build, and optimize scalable enterprise data platforms using Databricks Data Warehouse on the firm’s Azure Cloud platform. The role combines deep Databricks expertise with broad knowledge of the Microsoft Azure ecosystem to deliver high-performance data engineering, analytics, and data science solutions. It requires hands-on experience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, Azure Databricks, and Azure Synapse Data Warehouse. The manager will guide teams in delivering high-quality, robust data solutions that support actionable insights and decision-making.

The position is based at the firm’s corporate Pittsburgh, Pennsylvania location and follows a hybrid schedule of three days in the office and two days remotely from home. The Data Platform Manager leads and mentors data engineers, oversees scalable batch and streaming pipelines, establishes data governance and quality practices, and collaborates with stakeholders and analytics teams. The role also covers Lakehouse architecture, platform integration, security, performance optimization, and operational support.

Requirements

• 5–7+ years of hands-on data engineering or architecture experience, including at least 2–4 years specifically focused on Azure Databricks and Azure cloud technologies.
• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Deep proficiency in Python, specifically PySpark, SQL, PowerShell, and Scala.
• Proficiency with relational SQL and NoSQL databases, including document, key-value, graph, and columnar databases.
• Knowledge of data modeling and schema design principles.
• Knowledge of distributed systems and frameworks such as Apache Hadoop, Spark, or Presto/Trino for processing large data volumes.
• Understanding of Parquet, Avro, or ORC file formats and compression techniques.
• Hands-on experience with Azure Cloud infrastructure, including VNET networking, Key Vault, and identity management.
• Deep knowledge of Apache Spark runtime internals.
• Experience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, Azure Databricks, and Azure Synapse Data Warehouse.

Nice To Haves

• 2–5 years of experience managing a team of data engineers, data scientists, and/or analysts.
• Microsoft Certified: Azure Data Engineer Associate (DP-203).
• Databricks Certified Data Engineer Professional.
• Azure Solutions Architect Expert certification.
• Knowledge of MLflow for MLOps and orchestration tools such as Airflow.

Responsibilities

• Lead and mentor a team of data engineers, conduct code reviews, and ensure development standards.
• Support troubleshooting and incident management for data-related production issues.
• Collaborate with business stakeholders, data scientists, and other team members to gather requirements and translate them into technical specifications.
• Lead the design, development, and deployment of scalable, high-performance data pipelines using Azure Databricks.
• Ensure data integrity and availability, and efficiently extract, transform, and load data from various sources into the firm’s Azure Databricks Data Warehouse.
• Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights.
• Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
• Define best practices for data ingestion, processing, storage, and governance.
• Implement data quality checks and validation procedures to ensure data accuracy and integrity across APIs, databases, and streaming platforms.
• Collaborate with data scientists and analysts to operationalize and deploy machine learning models.
• Define end-to-end Lakehouse architecture using Delta Lake and implement the Bronze, Silver, and Gold medallion architecture layers.
• Oversee robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL, with minimal latency.
• Implement data transformations, enrichment, and quality checks using PySpark or Scala in Databricks.
• Integrate real-time and batch data sources using Apache Kafka and Azure Data Factory.
• Support large-scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory.
• Implement Unity Catalog for unified governance, data security, fine-grained role-based access control, privacy measures, and data lineage tracking.
• Tune Spark jobs and Databricks clusters to maximize throughput and maintain cost efficiency through autoscaling and cluster policies.
• Apply indexing strategies, query optimization, execution plans, and partitioning or sharding.
• Integrate Databricks with Azure services such as Azure Data Factory, Azure Data Lake Storage Gen2, and Azure DevOps for CI/CD pipelines.
• Develop and maintain data models and schemas for analysis and reporting.

Benefits

• Medical, dental, and vision insurance.
• 401K Program.
• Retirement Savings Program.
• Generous Paid Time Off.
• Paid holidays, including a floating holiday.
• WorkWell wellness program.
• Free use of the building gym.
• Caregiving assistance with Bright Horizons for child, elder, and pet care.
• Firm-wide emergency assistance fund.
• Free full access to LinkedIn Learning.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior