Lakehouse Architect/Databricks SME

CACI International • Washington, DC

4d • $103,800 - $218,100

About The Position

CACI is seeking an expert Lakehouse Architect / Databricks Subject Matter Expert to support the Department of Homeland Security (DHS) Office of the Inspector General (OIG). The role will architect and implement a modern enterprise data platform to help federal oversight professionals access, analyze, and derive insights from investigative and operational data.

The position leads the design and implementation of a secure, scalable data lakehouse using Azure Databricks and Azure Data Lake as the authoritative data foundation for AI, advanced analytics, and investigative workflows. The architect will design tiered storage zones, data catalog integration, and compute configurations for varied analytical workloads, and modernize analyst workflows to collaborative, cloud-based environments. The platform will support relational, graph, and geospatial data models, machine learning infrastructure, and secure Azure Government environments.

Requirements

• Bachelor's degree and 15 years of experience in data architecture, data engineering, analytics engineering, or a related field; stated equivalencies are a master's degree and 12 years, 21 years with no degree, or an associate degree and 17 years.
• U.S. citizenship and ability to obtain a Public Trust clearance and additional accesses as required.
• Deep Azure Databricks expertise, including demonstrated experience architecting and implementing lakehouse solutions with Delta Lake, Unity Catalog, and medallion architecture patterns.
• Proven experience designing enterprise data platforms for SQL analytics, data science notebooks using Python, R, or Scala, machine learning, and visualization integration.
• Strong data modeling background across dimensional, relational, graph, and data lake/lakehouse paradigms.
• Extensive hands-on experience with Azure Data Lake Storage, PySpark, SQL, performance optimization, compute configuration, and cloud analytics cost management.

Nice To Haves

• Experience implementing Databricks in Azure Government or other secure government cloud environments, such as AWS GovCloud or secure enclaves, and understanding of FedRAMP, NIST 800-53, and federal security requirements.
• Hands-on experience integrating Databricks with Azure AI services, MLflow, and feature engineering frameworks, and supporting MLOps and AI/ML model pipelines.
• Background in federal government, law enforcement, investigative analytics, or national security data environments, with understanding of sensitive data handling, data governance, and oversight mission analytics workflows.

Responsibilities

• Lead design and implementation of a secure, scalable enterprise data lakehouse using Azure Databricks, Azure Data Lake Storage, Delta Lake, and Unity Catalog.
• Architect tiered raw/bronze, curated/silver, and analytics-ready/gold storage zones, with data quality, transformation, and governance controls at each layer.
• Design workspace architecture, compute cluster strategies, autoscaling policies, job orchestration, and cost optimization for analytical and AI/ML workloads.
• Design data catalog integration, metadata management, and SQL-accessible tables and views to support governed discovery, lineage tracking, and self-service analytics.
• Support relational data warehousing, graph/network analysis, geospatial analytics, and machine learning feature engineering use cases.
• Modernize analyst and data scientist workflows from local workstation processing to collaborative cloud-based notebooks, SQL analytics, and platform-native tools.
• Provide technical expertise and surge support for data architecture challenges, analytical database design, performance optimization, and specialized data analysis.
• Design and implement machine learning engineering infrastructure, including MLflow integration, feature stores, model training environments, and integration patterns between Databricks and AI platform services.
• Develop data platform training content, best-practices guidance, workflow documentation, and user enablement materials for investigators, analysts, auditors, and data scientists.
• Collaborate with Government stakeholders, data engineers, AI engineers, and governance specialists to align architecture with mission analytics requirements, governance policies, and Agile delivery processes.

Benefits

• Flexible time off benefit.
• Learning resources.
• Healthcare, wellness, financial, retirement, family support, continuing education, and time off benefits.
• Competitive compensation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior