Platform Architect

Cooley AI•Washington, DC

2d•$175,000 - $250,000

About The Position

Cooley AI, the wholly-owned subsidiary of Cooley LLP, is seeking a Platform Architect to join its Innovation team. Cooley AI’s mission is to bring leading technology practices into the legal domain and help the firm and legal industry enter the AI era. Cooley AI employees are Cooley employees seconded to Cooley AI. The Platform Architect is a technical individual contributor in the Platform & Operations function, responsible for the architecture, design, and evolution of the Databricks platform underlying the Northstar Lakehouse. The role owns architecture decisions concerning platform organization, security, monitoring, and scale, including Unity Catalog metastore design, compute strategy, workspace configuration, CI/CD framework design, and adoption of Databricks capabilities. The architect works closely with the Director of Data Engineering & Platform and collaborates with Data Engineering to ensure decisions are implementable, performant, and aligned with the firm’s data and AI goals. The role owns Databricks platform architecture and governance, compute standards, platform capability evaluations, observability architecture, and living architecture documentation. It also evolves CI/CD and infrastructure-as-code practices, guides deployment standards, collaborates with engineering, governance, and Technology teams, and provides architectural guidance to platform operations.

Requirements

After orientation at Cooley LLP, exhibit proficiency in Microsoft Office, iManage, and other firm applications.
Ability to work extended and/or weekend hours as required.
Ability to travel as required.
At least 4 years of directly applicable experience in data platform engineering, cloud infrastructure, or data architecture, with hands-on responsibility for Databricks platform design and administration in a production enterprise environment.
Deep Databricks expertise, including Unity Catalog architecture and governance, Delta Lake table design and optimization, compute strategy, workspace administration, and enterprise-scale platform administration.
Strong CI/CD experience, including GitHub Actions, Databricks Asset Bundles, infrastructure-as-code patterns, and environment promotion pipeline design for a data engineering team.
Experience implementing Unity Catalog access controls, including catalog and schema privileges, row-level security, column-level masking, and audit logging.
Working knowledge of AWS infrastructure fundamentals, including IAM, VPC networking, S3 configuration, and connectivity between cloud infrastructure and Databricks workspaces.
Strong Python and SQL proficiency for platform automation, monitoring scripts, and pipeline architecture review.
Experience with Databricks Lakehouse Monitoring or a comparable data observability platform for drift detection and quality metric alerting.
Comfort using GitHub for version control, CI/CD pipeline development, and infrastructure-as-code management.
Strong written communication skills for architecture documentation, decision records, and technical standards.
Architectural judgment to make platform design decisions that serve long-term platform needs.
Ability to document architectural decisions, maintain current platform documentation, and preserve institutional knowledge in artifacts.
Ability to collaborate across data engineering, governance, and IT infrastructure functions and explain architecture decisions to non-platform audiences.
High standards for CI/CD quality, infrastructure-as-code discipline, and reproducible platform configuration.
Growth-oriented mindset and curiosity about Databricks capabilities, with the ability to assess features against platform needs.
Independent judgment to make sound architecture decisions within scope and escalate decisions requiring Director input.
Strong attention to detail and a high level of professionalism.
Unwavering ability to handle and maintain confidentiality regarding firm information, projects, and client data.

Nice To Haves

Bachelor’s degree.
Experience with Databricks FMAPI, Mosaic AI, Databricks Model Serving, or Databricks Apps and the architecture considerations for hosting AI workloads.
Familiarity with Apache Kafka, Qlik Replicate, or comparable CDC ingestion tools and Databricks real-time ingestion patterns.
Experience with Terraform, AWS CloudFormation, or comparable infrastructure-as-code tools for Databricks workspace and cloud resource management.
Familiarity with Data Vault 2.0 pipeline architecture and platform design considerations for supporting DV load patterns at scale.
Legal sector, professional services, or financial services experience involving data sensitivity, complex access controls, and compliance in a regulated environment.
Experience with Databricks Lakebase or a comparable transactional data layer for operational application database workloads on a lakehouse platform.
Experience using AI-assisted development tools such as Claude Code or GitHub Copilot for platform automation, architecture documentation, and infrastructure-as-code development.

Responsibilities

Own and document the Databricks platform architecture for the Northstar Lakehouse, including Unity Catalog metastore configuration, catalog and schema design, workspace organization, compute strategy, and Delta Lake table property standards.
Design and maintain Unity Catalog governance architecture, including catalog hierarchy, schema patterns, access controls, row-level security, column-level masking, and audit logging standards.
Define and maintain compute architecture standards for job clusters, SQL warehouses, Photon, instance pools, and cost-efficient compute patterns.
Evaluate Databricks capabilities such as FMAPI, Mosaic AI, Databricks Apps, Genie, Databricks Lakebase, and model serving, and recommend adoption timing and integration patterns.
Design data observability architecture, including Lakehouse Monitoring, alert thresholds, drift detection standards, and integration with the Data Quality Engineer’s DQE monitoring framework.
Maintain current platform architecture documentation, recording decisions, rationale, and implications for future platform evolution.
Own and evolve the Databricks CI/CD framework for automated promotion of pipeline code, notebook changes, and platform configuration through development, UAT, and production.
Maintain and extend Databricks Asset Bundles, GitHub Actions workflows, and environment-specific configuration management.
Define and enforce branching strategies, pull request standards, code review requirements, and deployment gates across Data Engineering and Platform & Operations.
Design and implement infrastructure-as-code patterns for version-controlled, repeatable, and recoverable Databricks resource management.
Collaborate with the DataOps Engineer on CI/CD operations and pipeline deployment automation, providing architectural direction as the platform scales.
Partner with the Director of Data Engineering & Platform on platform configuration, capability adoption, and infrastructure investment decisions.
Collaborate with Data Engineering on pipeline architecture decisions affecting the platform, including Delta Lake table properties, Spark patterns, Unity Catalog organization, and medallion conventions.
Work with Platform Governance & Quality on governance configuration, data classification, access controls, and audit logging.
Coordinate with the firm’s Technology function on external infrastructure dependencies, including network connectivity, gateway configuration, VM management, and cloud infrastructure.
Provide technical guidance to the Platform Operations Engineer on operational patterns, monitoring, and incident resolution requiring architectural context.
Perform all other duties as assigned or required.