Senior Data Platform Engineer

Freshfields•Raleigh, NC

About The Position

The Senior Data Platform Engineer is responsible for the day-to-day engineering and technical management of the Data Office platform ecosystem. The role manages and evolves cloud-native platforms across Microsoft Azure, Databricks, Unity Catalog, integration and ingestion services, Collibra, and Immuta, maintaining their reliability, availability, security, performance, and cost efficiency. It includes platform configuration and upgrades, automation, monitoring, access management, capacity and workload management, and incident resolution. The role also delivers senior-level data engineering outcomes across ingestion, integration, transformation, orchestration, governance, and data access. The engineer designs, builds, and maintains secure, scalable lakehouse solutions and production-grade pipelines that support analytics, business intelligence, AI, and machine learning. Working with architecture, engineering, governance, security, operations, and business teams, the engineer helps ensure data products are reliable, discoverable, controlled, and fit for purpose, while providing technical leadership and promoting a strong developer and consumer experience.

Requirements

5–10 years of hands-on experience in data engineering, data platform engineering, cloud engineering, or a closely related field.
Proven experience managing and supporting enterprise data platforms in production, including performance, workloads, storage, compute, access, availability, and cost.
Proven experience designing and implementing cloud data lake or lakehouse architectures at scale.
Experience operating in a Microsoft Azure enterprise environment and working with cross-functional platform, security, governance, and engineering teams.
Experience in a regulated environment, preferably legal or financial services, with an understanding of security, privacy, audit, and compliance obligations.
Advanced hands-on Databricks experience, including Spark, PySpark, Delta Lake, SQL, Workflows, clusters, SQL warehouses, jobs, performance tuning, and cost optimization.
Hands-on experience with Unity Catalog, including catalogs, schemas, permissions, lineage, secure data access, and governance administration.
Strong experience with Microsoft Azure services, including Azure Data Lake Storage Gen2, Azure Data Factory, Azure Functions, Logic Apps, Key Vault, networking, identity, monitoring, and related services.
Strong Python and SQL skills and the ability to develop, review, troubleshoot, and optimize production-grade code and queries.
Experience designing and supporting integration and ingestion patterns across databases, enterprise applications, files, APIs, and streaming or event-based sources.
Experience with Collibra or a comparable data catalog and governance platform, including metadata, lineage, stewardship, ownership, and workflow integration.
Experience with Immuta or a comparable data security platform, including policy-based access, sensitive data controls, auditability, and Databricks integration.
Knowledge of identity and access management, role-based and attribute-based access control, service principals, managed identities, secrets, and least-privilege design.
Experience with platform monitoring, observability, capacity management, incident response, root-cause analysis, resilience, backup, and recovery.
Experience with CI/CD, infrastructure as code, source control, automated testing, and release management using Terraform, Git, Azure DevOps, GitHub Actions, or equivalent tools.
Strong analytical, problem-solving, troubleshooting, and root-cause analysis skills.
Ability to balance platform stability and control with delivery speed, usability, and innovation.
Clear verbal and written communication skills and ability to work effectively across technical and non-technical teams.
Ability to prioritize operational work, engineering delivery, platform improvements, and technical debt.
Proactive ownership, sound judgement, attention to detail, and commitment to continuous improvement and knowledge sharing.

Nice To Haves

Experience with dbt, Apache Airflow, Azure Event Hubs, Spark Structured Streaming, or comparable transformation and orchestration technologies.
Experience with Azure Synapse Analytics, Microsoft Purview, containerization, Kubernetes, or enterprise API management.
Knowledge of FinOps practices for data platforms, including attribution, budgeting, forecasting, chargeback or showback, and cost-control policies.
Understanding of data quality, metadata management, lineage, retention, classification, privacy, and records-management controls.
Familiarity with agile delivery and IT service management practices, including change, incident, problem, and release management.

Responsibilities

Manage the Data Office platform estate across Azure, Databricks, Unity Catalog, integration and ingestion services, Collibra, and Immuta, ensuring reliability and alignment with enterprise standards.
Monitor and optimize platform performance, workloads, clusters, SQL warehouses, jobs, storage, and compute for reliability, scalability, user experience, and cost efficiency.
Own capacity and workload management, including resource allocation, concurrency, job scheduling, autoscaling, workload isolation, and evaluation of serverless and cluster-based compute.
Administer platform configuration, workspaces, environments, service connections, secrets, policies, and technical integrations across development, test, and production.
Manage data access controls and entitlements across Azure, Databricks, Unity Catalog, Collibra, and Immuta using least-privilege, role-based, attribute-based, and policy-based principles.
Implement and maintain Unity Catalog structures, including catalogs, schemas, permissions, data objects, lineage, and controlled data sharing.
Support Collibra integration and adoption for cataloging, metadata, business glossaries, lineage, ownership, stewardship, and governance workflows.
Design, build, and maintain secure, scalable lakehouse solutions using Azure Data Lake Storage, Databricks, Delta Lake, Spark, PySpark, Python, and SQL.
Develop and optimize batch and streaming ingestion, integration, ETL, and ELT pipelines using Azure Data Factory, Databricks Workflows, APIs, event-based services, and suitable orchestration tools.
Create reusable integration and ingestion patterns for structured, semi-structured, and unstructured data from enterprise applications, files, databases, APIs, and event streams.
Implement platform and pipeline observability, including monitoring, logging, alerting, performance and usage metrics, cost reporting, failure analysis, and operational dashboards.
Lead technical incident investigation, root-cause analysis, remediation, and problem management for platform, workload, access, and pipeline issues.
Automate platform deployment and configuration through infrastructure as code, CI/CD, testing, release management, and version control, using tools such as Terraform, Git, Azure DevOps, or GitHub Actions.
Implement data quality, reconciliation, validation, exception handling, recovery, and audit controls for reliable and trusted data products.
Plan and execute platform upgrades, feature adoption, migrations, configuration changes, resilience testing, backup and recovery activities, and lifecycle management.
Maintain technical standards, runbooks, architecture diagrams, support procedures, data dictionaries, operating documentation, and knowledge articles.
Provide technical leadership, coaching, design review, and best-practice guidance to data engineers and platform users.
Collaborate with architecture, cloud operations, cyber security, risk, governance, analytics, and business teams to translate requirements into secure and sustainable engineering solutions.