Databricks Architect (Lead Data Platform Architect)

iLink Digital • New York, NY

1d

About The Position

We are looking for a highly skilled Databricks Architect to design, build, and scale enterprise-grade Lakehouse data platforms. This role will drive architecture strategy, platform standardization, and enterprise data modernization initiatives, leveraging Databricks and cloud ecosystems. The ideal candidate brings deep expertise in Spark, Delta Lake, and cloud-native architecture, along with strong leadership in driving large-scale data transformations.

Requirements

• 12+ years of experience in data engineering, architecture, or platform design.
• 5+ years of hands-on experience with Databricks (must-have).
• 5+ years of hands-on experience with Apache Spark / PySpark / SQL.
• Strong expertise in Delta Lake.
• Strong expertise in Distributed data processing.
• Experience with at least one cloud: Azure (preferred), AWS, or GCP.

Responsibilities

• Define and implement end-to-end Databricks Lakehouse architecture.
• Design scalable systems for batch & real-time data processing and structured & unstructured workloads.
• Establish medallion architecture (Bronze, Silver, Gold layers) as a standard.
• Lead deployment and optimization of Azure Databricks / AWS Databricks / GCP Databricks.
• Define standards for workspace design & cluster strategy, job orchestration, and data storage (Delta Lake).
• Drive adoption of Unity Catalog, MLflow, Databricks SQL & Photon.
• Architect robust data ingestion frameworks: Batch (ADF, Airflow) and Streaming (Kafka, Event Hub).
• Define reusable patterns for ETL/ELT pipelines and data modeling (star schema, data vault, dimensional models).
• Guide engineering teams on best practices in Spark/PySpark optimization.
• Optimize workloads for query performance, cluster utilization, and storage efficiency.
• Implement cost governance strategies (auto-scaling, job clusters, spot instances).
• Architect enterprise-grade governance frameworks: Data lineage, cataloging, metadata management, and fine-grained access control (RBAC/ABAC).
• Ensure compliance with data privacy and regulatory standards.
• Integrate Databricks with data sources (ERP, CRM, APIs, IoT), BI tools (Power BI, Tableau), and ML pipelines and AI platforms.
• Collaborate with cloud architects for networking, security, and storage strategies.
• Provide architectural guidance to data engineers, scientists, and TPMs.
• Conduct design reviews and enforce architecture governance.
• Mentor teams on emerging patterns: Data Mesh, DataOps / MLOps, GenAI workloads on Databricks.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed