Lakehouse Architect/Databricks SME

CACI•Washington, DC

About The Position

CACI is seeking an expert Lakehouse Architect / Databricks Subject Matter Expert to support the Department of Homeland Security (DHS) Office of the Inspector General (OIG). The role involves architecting and implementing a modern enterprise data platform to support access to, analysis of, and insight from investigative and operational data. The position leads the design and implementation of a secure, scalable lakehouse using Azure Databricks and Azure Data Lake as an authoritative data foundation for AI, advanced analytics, and investigative workflows. It includes designing tiered storage zones, integrating data catalogs, configuring compute for varied workloads, and moving analysts from local desktop processing to collaborative cloud-based tools. The architecture supports relational, graph, and geospatial data models, machine learning infrastructure, and secure Azure Government environments.

Requirements

Bachelor's degree and 15 years of experience in data architecture, data engineering, analytics engineering, or a related field; stated alternatives are a Master's degree and 12 years, 21 years with no degree, or an associate degree and 17 years.
U.S. citizenship and the ability to obtain a Public Trust clearance and additional accesses as required.
Deep expertise in Azure Databricks, with demonstrated experience architecting and implementing lakehouse solutions using Delta Lake, Unity Catalog, and medallion architecture patterns.
Experience designing enterprise data platforms for diverse analytical workloads, including SQL analytics, Python, R, and Scala data science notebooks, machine learning, and visualization integration.
Strong data modeling background across dimensional modeling, relational databases, graph databases, and data lake/lakehouse architectures.
Extensive hands-on experience with Azure Data Lake Storage, PySpark, SQL, performance optimization, compute configuration, and cloud analytics platform cost management.

Nice To Haves

Experience implementing Databricks in Azure Government or other secure government cloud environments, such as AWS GovCloud or secure enclaves, and understanding of FedRAMP, NIST 800-53, and federal security requirements.
Hands-on experience integrating Databricks with Azure AI services, MLflow, and feature engineering frameworks, and supporting MLOps and AI/ML model pipelines.
Background in federal government, law enforcement, investigative analytics, or national security data environments, including understanding of sensitive data handling, data governance, and oversight mission analytical workflows.

Responsibilities

Lead the design and implementation of a secure, scalable enterprise data lakehouse using Azure Databricks, Azure Data Lake Storage, Delta Lake, and Unity Catalog.
Architect raw/bronze, curated/silver, and analytics-ready/gold storage zones, with data quality, transformation, and governance controls.
Design workspace architecture, compute cluster strategies, autoscaling policies, job orchestration, and cost optimization for analytical and AI/ML workloads.
Design data catalog integration and metadata management, and provide SQL-accessible tables and views for governed discovery, lineage, and self-service analytics.
Support relational data warehousing, graph/network analysis, geospatial analytics, and machine learning feature engineering.
Modernize analyst and data scientist workflows from local workstation processing to collaborative cloud-based notebooks, SQL analytics, and platform-native tools.
Provide technical expertise and surge support for data architecture challenges, analytical database design, performance optimization, and specialized data analysis.
Design and implement machine learning engineering infrastructure, including MLflow integration, feature stores, model training environments, and integration patterns with AI platform services.
Develop training content, best-practice guidance, workflow documentation, and user enablement materials for investigators, analysts, auditors, and data scientists.
Collaborate with Government stakeholders, data engineers, AI engineers, and governance specialists to align architecture with mission analytics requirements, governance policies, and Agile delivery processes.