Director of Data Engineering & Platform

Cooley•San Francisco, CA

About The Position

Cooley is seeking a Director of Data Engineering & Platform to join the Data team within the Innovation department. As a leading technology law firm, Cooley is determined to become a leader in the digital practice of law. The Director of Data Engineering & Platform is a leadership role responsible for building, leading, and operating the Data Engineering & Platform function across Data Engineering and Platform & Operations. This role joins the Northstar Lakehouse program at a critical build phase, inheriting a live platform that is actively moving data into the Silver layer, with core Data Vault 2.0 structures being established across the firm's primary legal sector entities. The Director will develop high-performing Data Engineering and Platform Operations teams that deliver the Northstar Lakehouse from Silver layer foundations to gold layer products serving the firm’s AI and analytics ambitions. The platform is built on Databricks on AWS with a medallion architecture, Data Vault 2.0 methodology for non-financial data domains, and a two-track gold layer serving both legal intelligence and financial reporting consumption patterns. This is a hands-on leadership role that will build a greenfield data platform moving quickly, building correctly, and delivering at the pace a legal AI program demands.

Requirements

After orientation at Cooley LLP, exhibit proficiency in the Microsoft Office suite, iManage, and other firm applications.
Ability to work extended and/or weekend hours, as required.
Ability to travel, as required.
10+ years of experience in data engineering, data platform architecture, or enterprise solution architecture with demonstrated leadership of a data engineering function in a cloud-native environment with 5+ years of management experience in relevant roles.
Proven experience building a greenfield cloud data platform from the ground up, including hands-on involvement in medallion architecture design, Data Vault 2.0 implementation, pipeline framework establishment, and gold layer consumption pattern development.
Deep Databricks expertise including Unity Catalog configuration and governance, Delta Lake table design and optimization, Databricks SQL and job cluster architecture, compute tier strategy, workspace organization, and Databricks platform administration at an enterprise scale.
Strong Data Vault 2.0 literacy, sufficient to assess whether Raw Vault hub, link, and satellite implementations are methodologically correct.
Experience designing and delivering a two-track or multi-layer gold layer architecture serving both analytical BI consumption and AI-powered product consumption from the same Silver layer foundations.
Demonstrated experience directing ESA, contract, and transitional resources alongside a permanent engineering team, setting clear work agendas, managing delivery accountability, and making informed decisions about resource retention and transition.
Proven track record of effective leadership.
Experience owning cross-functional infrastructure relationships, specifically coordinating between a data platform engineering team and an IT infrastructure or enterprise technology function on network, gateway, VM, and cloud connectivity dependencies.
Proven Agile delivery leadership in a squad-based data platform program, including sprint planning, EPIC delivery management, and definition of done discipline across multiple parallel data domain workstreams.
Comfort using GitHub for version control, CI/CD pipeline oversight, and engineering standards governance in a collaborative data platform environment.
Bachelor’s degree.

Nice To Haves

Legal sector, professional services, or financial services experience with understanding of complex operational data environments including matter management, client relationship data, timekeeper and billing data, or comparable professional services data complexity.
Experience with 3NF relational modeling for operational or ERP systems, specifically in the context of building a financial or operational reporting gold layer alongside a Data Vault analytical layer.
Experience with dbt for transformation layer implementation including model development, testing frameworks, documentation standards, and the governance of dbt as a platform-wide transformation standard.
Familiarity with Qlik Replicate, AWS DMS, or comparable CDC ingestion tooling for real-time or near-real-time data replication from operational source systems into a cloud data platform.
Experience with Informatica MDM, DQE, or comparable enterprise master data and data quality platforms and how they integrate with a Databricks lakehouse environment.
Familiarity with infrastructure-as-code tooling such as Terraform, AWS CloudFormation, or comparable for Databricks workspace and resource management.
AI-assisted development fluency including Claude Code, GitHub Copilot, or comparable tools used to accelerate engineering delivery, code review, and documentation workflows across the team.
Experience designing or operating a data platform that serves as the foundation for RAG pipelines, LLM fine-tuning datasets, vector search indexes, or comparable AI infrastructure.

Responsibilities

Lead and develop the Innovation Data Engineering & Platform function across Data Engineering and Platform & Operations.
Build a high-performing, accountable team through clear expectations, consistent feedback.
Provide direct leadership to the team and as the function matures and team members develop, make deliberate decisions regarding introducing a management layer beneath the Director.
Direct contract resources including transitional Enterprise Solution Architects and Contractor program resources, setting clear work agendas, managing delivery accountability, and making recommendations on resource extension or transition based on platform needs and team capability development.
Partner with the Director of Data on function strategy, hiring sequencing, budget, and the long-term capability roadmap for the Data Engineering & Platform function as the Northstar Lakehouse transitions from build phase to operational maturity.
Develop the Senior Data Engineering Manager as the technical engineering leader this function needs over time. Provide direct, frequent feedback on delivery quality, team management, and technical decision-making.
Own workforce planning for the function, including permanent hire sequencing, contractor-to-permanent conversion decisions, and the right-sizing of the team as the platform moves from construction to steady-state operations.
Serve as direct supervisor and mentor to direct reports.
Provide day-to-day supervision of direct reports, ensure compliance with assigned work hours and monitor for compliance with all firm and department policies.
Manage staffing coverage, review and process time logs/time off requests.
Support business professional development and continued educational opportunities.
In collaboration with immediate supervisor and central HR, participate in hiring, performance appraisals, counseling, termination and other employee lifecycle events.
Own the technical delivery of the Northstar Lakehouse build across the Bronze, Silver, and Gold layers of the Databricks medallion architecture.
Drive reliable, high-quality data pipeline delivery across Data Vault 2.0 structures for non-financial legal sector domains and 3NF relational structures for the 3E financial and billing data domain.
Establish and maintain the two-track gold layer architecture: a Data Vault Business Vault consumption layer serving legal intelligence, client and matter analytics, competitive intelligence, and AI-powered legal products, and a 3NF relational reporting layer serving financial reporting, billing analytics, and operational dashboards from the 3E legal billing and matter management system.
Ensure the Data Engineering team implements Data Vault 2.0 pipelines in alignment with the methodology standards defined by the Principal Data Architect in the Platform Governance & Quality function.
Create the conditions for the Principal Data Architect to provide effective methodology oversight, code review, and standards enforcement without friction.
Drive the Bronze layer ingestion quality baseline, ensuring source-to-platform data completeness, reconciliation back to source systems, and structural validation are established across all active data sources before Silver layer certification is approved.
Own the delivery cadence of the Data Engineering function within the squad-based EPIC delivery model. Ensure Agile ceremonies are effective, sprint commitments are realistic and met, and the engineering team is unblocked and productive across active Data Domain Squads.
Partner with the Director of Data Platform Governance & Quality on the certification workflow for Bronze and Silver layer data assets, ensuring data quality gates, governance controls, and access policies are embedded in the engineering build process rather than applied as post-delivery audits.
Own the Databricks platform architecture decisions for the Northstar Lakehouse, including Unity Catalog metastore configuration, compute tier design, job cluster and SQL warehouse architecture, Delta Lake table property standards, workspace organization, and platform security configuration.
Direct the Platform Architect in maintaining and evolving the Databricks platform as capabilities change, new Databricks features become available, and the platform's consumption patterns grow. Ensure the platform architecture stays current, performant, and aligned with the firm's AI and analytics ambitions.
Own the CI/CD framework for data pipeline deployment, ensuring reliable and automated promotion of pipeline code through development, UAT, and production environments. Hold the DataOps Engineer accountable for CI/CD framework operation and evolution as the pipeline estate scales.
Oversee Platform & Operations reliability across the live Databricks environment, including compute health, pipeline execution monitoring, incident management, and escalation protocols across a he platform that is live and serving active data consumers. Operational reliability is a must from Day 1.
Manage the integration of external tools and data sources with the Databricks platform, including Qlik Replicate for CDC ingestion, gateway and VM infrastructure coordination, and any additional tooling integrations required as the platform scales.
Own the relationship with the firm's Technology function as the senior accountability point for all infrastructure dependencies that sit outside the Databricks platform. This includes network connectivity, gateway configuration, VM management, data center and cloud infrastructure coordination, and security policy alignment between the platform and the broader technical environment.
Work closely with the Technology function on infrastructure dependencies for the Data Engineering & Platform team. Proactively surface infrastructure requirements early, build credible relationships with Technology counterparts, and resolve cross-functional blockers at the appropriate level without escalating unnecessarily.
Partner with the Director of Data Platform Governance & Quality on cross-functional platform standards including data certification workflows, Unity Catalog governance configuration, access control policies, and the technical governance standards the Platform Governance & Quality function defines and the Engineering team implements.
Collaborate with the Director of Data Products and the Intelligence & Client Services function on gold layer data product delivery, ensuring the engineering team understands the analytical and AI consumption requirements that drive gold layer design decisions and delivers the data structures those products depend on.
Engage with the Data Council through the Data Platform Governance & Quality function, providing engineering perspective on platform capability, delivery timelines, and technical feasibility of governance-driven data requirements surfaced by Data Domain Squads and working groups.
All other duties as assigned or required.