Databricks Solution Architect

EXL • New Jersey, NJ

19h

About The Position

Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, with strong focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems. Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers to support reporting, advanced analytics, ML, and business-critical decisioning use cases. Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables for batch and near real-time processing. Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management across Databricks solutions. Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and integration with enterprise governance services such as Purview. Optimize solution performance by tuning Spark workloads, cluster policies, partitioning strategy, file sizing, caching, and compute cost management for large-scale data processing. Collaborate with business stakeholders, product owners, analysts, architects, and downstream consumers to translate functional and non-functional requirements into scalable technical designs. Provide technical leadership to engineering teams by reviewing designs, guiding implementation, resolving architectural bottlenecks, and establishing best practices for Databricks-based delivery. Evaluate and recommend Databricks capabilities such as Photon, serverless compute, Lakehouse Federation, and streaming patterns to improve scalability, maintainability, and time to value. Ensure strong delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution including sprint planning, backlog refinement, and design reviews.

Requirements

• Databricks Lakehouse
• Delta Lake
• PySpark
• SQL
• Databricks Workflows
• Auto Loader
• Delta Live Tables
• Unity Catalog
• RBAC/ABAC controls
• Purview
• Spark workloads
• Agile execution

Nice To Haves

• Photon
• serverless compute
• Lakehouse Federation
• streaming patterns

Responsibilities

• Lead the end-to-end architecture and solution design for enterprise data platforms on Azure, with strong focus on Databricks Lakehouse, Delta Lake, and scalable cloud-native data ecosystems.
• Define target-state data architecture, ingestion patterns, transformation frameworks, and serving layers to support reporting, advanced analytics, ML, and business-critical decisioning use cases.
• Design and implement robust ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, Auto Loader, and Delta Live Tables for batch and near real-time processing.
• Own architecture standards for data modeling, medallion design, reusable engineering patterns, CI/CD, code quality, environment strategy, and release management across Databricks solutions.
• Drive platform governance and security using Unity Catalog, RBAC/ABAC controls, lineage, auditability, and integration with enterprise governance services such as Purview.
• Optimize solution performance by tuning Spark workloads, cluster policies, partitioning strategy, file sizing, caching, and compute cost management for large-scale data processing.
• Collaborate with business stakeholders, product owners, analysts, architects, and downstream consumers to translate functional and non-functional requirements into scalable technical designs.
• Provide technical leadership to engineering teams by reviewing designs, guiding implementation, resolving architectural bottlenecks, and establishing best practices for Databricks-based delivery.
• Evaluate and recommend Databricks capabilities such as Photon, serverless compute, Lakehouse Federation, and streaming patterns to improve scalability, maintainability, and time to value.
• Ensure strong delivery governance through estimation, technical planning, dependency management, risk mitigation, and Agile execution including sprint planning, backlog refinement, and design reviews.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed