Databricks on AWS Platform Architect

Stefanini Group•Dallas, TX

3d•Onsite

About The Position

We are seeking a skilled Databricks on AWS Platform Architect to design, build, and manage a secure, scalable, and highly available Databricks Lakehouse Platform on AWS. This role involves architecting data platforms using AWS S3, Delta Lake, and Unity Catalog for enterprise analytics and AI. You will define networking architecture, establish multi-environment strategies with Infrastructure as Code, and implement data governance, lineage, and compliance controls. The position requires designing high-performance data ingestion and processing architectures, enabling AI and GenAI capabilities, and defining CI/CD, monitoring, and observability frameworks. Additionally, you will optimize platform performance, reliability, and costs, and provide architectural leadership for Lakehouse modernization and enterprise data platform adoption.

Requirements

Databricks on AWS Platform Architecture experience
Experience with AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces
Knowledge of networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards
Experience with Terraform and Infrastructure as Code (IaC)
Experience with data governance, lineage, metadata management, and compliance controls
Experience designing data ingestion and processing architectures (batch, streaming, real-time)
Experience with AI and GenAI capabilities (Mosaic AI, Vector Search, Model Serving, Genie Spaces)
Experience with CI/CD, monitoring, logging, and observability frameworks (GitHub, Jenkins, CloudWatch, Databricks Workflows)
Experience with FinOps and workload optimization practices
Experience with Databricks Platform Administrator certification
Experience with Databricks Certified Data Engineer certification
Experience with AWS Certification (DevOps/Solution Architect)
Experience with Terraform
Core Technologies: Databricks, AWS S3, Delta Lake, Unity Catalog, Spark/PySpark, Terraform, GitHub, CI/CD, CloudWatch, Mosaic AI, Genie, Vector Search, MLflow, Kafka, Airflow, Kubernetes.

Responsibilities

Architect and manage a secure, scalable, and highly available Databricks Lakehouse Platform on AWS.
Design data platforms leveraging AWS S3, Delta Lake, Unity Catalog, and Databricks Workspaces for enterprise analytics and AI.
Define networking architecture including VPC, PrivateLink, IAM roles, security groups, and encryption standards.
Establish multi-environment strategies (Dev, Test, UAT, Prod) with automated provisioning through Terraform and Infrastructure as Code (IaC).
Implement enterprise-wide data governance, lineage, metadata management, and compliance controls using Unity Catalog.
Design high-performance data ingestion and processing architectures supporting batch, streaming, and real-time analytics workloads.
Enable AI and GenAI capabilities through Mosaic AI, Vector Search, Model Serving, AI/BI Dashboards, and Genie Spaces.
Define CI/CD, monitoring, logging, and observability frameworks using GitHub, Jenkins, CloudWatch, and Databricks Workflows.
Optimize platform performance, reliability, scalability, and cloud costs through FinOps and workload optimization practices.
Provide architectural leadership and best practices for Lakehouse modernization, advanced analytics, AI governance, and enterprise data platform adoption.