Databricks on AWS Platform Architect

Stefanini Group•Dallas, TX

3d•Onsite

About The Position

We are seeking a skilled Databricks on AWS Platform Architect to design, build, and manage a secure, scalable, and highly available Databricks Lakehouse Platform on AWS. This role involves architecting data platforms using AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces for enterprise analytics and AI. You will define networking architecture, establish multi-environment strategies with IaC, and implement enterprise-wide data governance and compliance controls. The position requires designing high-performance data ingestion and processing architectures, enabling AI/GenAI capabilities, and defining CI/CD, monitoring, and observability frameworks. Optimization of platform performance, reliability, scalability, and cloud costs is crucial, as is providing architectural leadership for Lakehouse modernization and enterprise data platform adoption.

Requirements

Databricks on AWS Platform Architecture experience
Experience with AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces
Knowledge of networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards
Experience with Terraform and Infrastructure as Code (IaC)
Experience with enterprise-wide data governance, lineage, metadata management, and compliance controls using Unity Catalog
Experience designing data ingestion and processing architectures (batch, streaming, real-time)
Experience with AI and GenAI capabilities (Mosaic AI, Vector Search, Model Serving, AI/BI Dashboards, Genie Spaces)
Experience with CI/CD, monitoring, logging, and observability frameworks (GitHub, Jenkins, CloudWatch, Databricks Workflows)
Experience with FinOps and workload optimization practices
Architectural leadership skills
Core Technologies: Databricks, AWS S3, Delta Lake, Unity Catalog, Spark/PySpark, Terraform, GitHub, CI/CD, CloudWatch, Mosaic AI, Genie, Vector Search, MLflow, Kafka, Airflow, Kubernetes
Certification: DataBricks Platform Administrator /Databricks Certified Data Engineer/ AWS Certification (DevOps/Solution Architect)
Terraform

Responsibilities

Architect and manage a secure, scalable, and highly available Databricks Lakehouse Platform on AWS.
Design data platforms leveraging AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces for enterprise analytics and AI.
Define networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards.
Establish multi-environment strategies (Dev, Test, UAT, Prod) with automated provisioning through Terraform and Infrastructure as Code (IaC).
Implement enterprise-wide data governance, lineage, metadata management, and compliance controls using Unity Catalog.
Design high-performance data ingestion and processing architectures supporting batch, streaming, and real-time analytics workloads.
Enable AI and GenAI capabilities through Mosaic AI, Vector Search, Model Serving, AI/BI Dashboards, and Genie Spaces.
Define CI/CD, monitoring, logging, and observability frameworks using GitHub, Jenkins, CloudWatch, and Databricks Workflows.
Optimize platform performance, reliability, scalability, and cloud costs through FinOps and workload optimization practices.
Provide architectural leadership and best practices for Lakehouse modernization, advanced analytics, AI governance, and enterprise data platform adoption.