Azure Databricks & Agentic AI Architect

TalentOla•Chicago, IL

1d•Hybrid

About The Position

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems. The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Requirements

Azure Databricks
Delta Lake
Unity Catalog
Azure Data Factory
Azure OpenAI
Knowledge Graph
RAG Architecture
LangChain
LangGraph
MCP Protocol
Vector Databases
AI Agent Orchestration
PySpark
Spark SQL
Python
SQL
ELT/ETL Modernization
Azure DevOps
GitHub Actions
CI/CD
MLOps
LLMOps
Evaluation Frameworks
AI Testing Frameworks
15+ years of Data & Analytics experience
Strong Azure Databricks expertise
Hands-on experience building Agentic AI platforms, AI-powered SDLC frameworks, and autonomous data engineering ecosystems.
Comfortable leading enterprise-scale AI transformation initiatives and engaging CXO stakeholders on AI strategy and business value.
Engineering Degree – BE/ME/BTech/MTech/BSc/MSc.
Azure
DataBricks
Agentic AI
ETL
SQL
Data Engineering

Nice To Haves

Databricks Certified Data Engineer Professional
Databricks Certified Solution Architect
Microsoft Azure Solution Architect (AZ-305)
Azure Data Engineer (DP-203)
Microsoft Applied Skills – Azure OpenAI
Generative AI / Agentic AI Certifications
Technical certification in multiple technologies is desirable.

Responsibilities

Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.
Define autonomous workflows using AI Agents for data ingestion, data mapping, schema evolution, data quality remediation, metadata enrichment, pipeline optimization, and root cause analysis.
Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.
Lead architecture for AI-enabled Data Development Lifecycle across requirement analysis, data modeling, pipeline generation, automated testing, code review, documentation, deployment, and monitoring.
Implement AI copilots to accelerate developer productivity.
Enable automated lineage creation and intelligent impact analysis.
Design scalable Lakehouse platforms using Azure Databricks, Delta Lake, Unity Catalog, ADLS Gen2, Databricks Workflows, and Delta Live Tables.
Architect RAG-based solutions using enterprise data assets.
Design agent orchestration frameworks using Azure OpenAI, LangGraph, Semantic Kernel, AutoGen, and MCP-enabled architectures.
Build domain-specific AI agents supporting Data Engineering and Analytics teams.
Define guardrails for enterprise GenAI adoption.
Implement prompt governance, observability, cost monitoring, auditability, explainability, and security controls.
Establish governance models for autonomous AI agents.
Design self-healing data pipelines.
Implement AI-driven incident triage, failure prediction, capacity planning, cost optimization, and SLA monitoring.
Enable intelligent workload placement and model routing.
Drive AI-First Data Engineering transformation.
Define enterprise patterns, accelerators, and reusable AI agents.
Mentor architects, data engineers, and AI engineers.
Lead executive conversations on AI adoption, ROI, and transformation roadmaps.