Experienced Director, Lead, Enterprise Data Architecture

Evercore • , NY

21h

About The Position

Evercore seeks to hire a Director, Lead, Enterprise Data Architecture to serve as the technical authority for data engineering standards and platform architecture within Global Technology’s AI, Data & Analytics organization. This role is responsible for defining and governing how data pipelines, data products, and AI workloads are architected, built, secured, and operated on the firm’s Databricks platform. This high-impact role sits at the intersection of platform engineering, distributed systems design, and developer experience. The ideal candidate will bring deep expertise in Databricks, Spark, enterprise data governance and event-driven architectures for regulated environments.

Requirements

• Minimum 10+ years of experience in data engineering, enterprise data platform architecture, or large-scale analytics platforms.
• Minimum 5+ years of hands-on experience designing and scaling Databricks and Spark-based platforms in enterprise environments.
• Deep expertise in Apache Spark (performance tuning, optimization strategies, structured streaming, distributed compute patterns).
• Experience architecting Azure cloud infrastructure.
• Proven experience architecting event-driven and streaming data systems using pub/sub technologies such as Kafka or equivalent platforms.
• Demonstrated experience designing enterprise-grade ETL/ELT frameworks and internal SDKs for standardized pipeline development and deployment.
• Strong experience implementing Unity Catalog or equivalent enterprise data governance tooling, including fine-grained access control models and regulatory audit readiness.
• Experience operating within regulated or highly controlled environments with formal compliance and audit requirements.
• Bachelor’s degree in Computer Science, Engineering, or related quantitative discipline required.
• Valid United States work authorization and not eligible for employer visa sponsorship.

Nice To Haves

• Advanced degree preferred.

Responsibilities

• Define and govern enterprise architecture standards for batch and streaming data pipelines on Azure Databricks, including Spark optimization, structured streaming, and pub/sub integration (e.g., Event Hub, Kafka).
• Design and maintain standardized internal SDKs and reusable frameworks for ETL development.
• Establish the data engineering SDLC, including schema evolution policies, artifact versioning, environment promotion controls and CI/CD standards.
• Design secure storage architectures leveraging Azure Data Lake Storage Gen2 (ADLS Gen2).
• Architect and administer secure, scalable Databricks environments, including workspace segmentation strategies, Unity Catalog design, role-based and attribute-based access controls, row/column-level security, and compute isolation policies.
• Design and implement enterprise metadata, observability, and data quality frameworks, including automated lineage capture, SLA monitoring, operational telemetry, and cost attribution models.
• Establish data product standards, including schema contract enforcement, versioning policies, backward compatibility requirements, domain ownership models, and publish/subscribe consumption patterns.
• Define best practices for exploratory and non-production workloads, including guardrails for data scientist experimentation, compute policy controls, secure data access boundaries, and promotion of validated assets into production pipelines.
• Partner with Data Scientists and Machine Learning Engineers to define the lifecycle from experimentation to production, including feature engineering standards, model registry governance, reproducibility controls, and approved deployment pathways.

Benefits

• Medical, prescription, dental, and vision insurance, including healthcare savings and reimbursements accounts
• 401(k) Retirement Plan
• Life and disability insurance, including additional voluntary financial protection insurance
• Well-being resources and programs, including mental health and mindfulness programs, digital wellness platforms, well-being events, and targeted on-site health services
• Family-building and family-support benefits
• Paid parental, caregiver, marriage and bereavement leave
• Commuter benefits, health club membership discounts, and other corporate discounts
• Paid holidays, vacation days, personal days, sick days, and volunteer opportunities

What This Job Offers

• Job Type: Full-time
• Career Level: Director
• Number of Employees: 501-1,000 employees