Databricks Architect

Confiz•Chicago, IL
•Hybrid

About The Position

Confiz is seeking a Databricks Architect in Chicago, Illinois, to lead the data-layer design and delivery for enterprise AI use cases and serve as the technical authority on the Databricks platform. The role owns the end-to-end architecture of the data foundation powering AI/BI solutions, from data discovery and readiness through secure, governed delivery, ensuring alignment with enterprise standards. The work model is hybrid, with 2–3 days onsite in Chicago. The Architect leads data-layer architecture and delivery for prioritized AI use cases, translates requirements into data models, pipelines, and Databricks platform patterns, and establishes data quality, lineage, observability, and reusable architecture standards. The role also facilitates data discovery with business stakeholders and data owners, assesses and resolves data readiness gaps, and maintains discovery artifacts and playbooks. The position architects the Databricks platform, including workspace topology, compute, storage, networking, CI/CD, environment strategy, and infrastructure-as-code practices. It advises on cost, performance, and capacity, and guides adoption of Databricks capabilities. The Architect explicitly owns end-to-end platform security design, including Unity Catalog governance, fine-grained access controls, PII protection for natural-language query experiences, and documentation of controls for security, privacy, compliance, and audit review. The role partners with the Lead AI/BI Engineer, provides technical direction and design reviews for data engineers, and communicates architecture decisions, trade-offs, and risks to technical teams and business stakeholders. Success includes delivering trusted data layers on schedule, resolving readiness issues early, preventing PII exposure through AI interfaces, and establishing a reusable Unity Catalog governance pattern.

Requirements

  • At least 8 years of experience in data architecture or data engineering.
  • At least 3 years architecting Databricks solutions in production environments.
  • Deep expertise with the Databricks Lakehouse platform, including Delta Lake, Unity Catalog, Databricks SQL, workflows/jobs, and medallion architectures.
  • Demonstrated ownership of data security and governance design, including Unity Catalog governance models, row- and column-level security, and data masking.
  • Hands-on experience with PII classification and protection strategies, ideally for AI, natural-language query, or conversational analytics workloads.
  • Strong data modeling skills, including dimensional, data vault, or domain-driven data product design.
  • Proficiency in SQL and Python (PySpark).
  • Experience with at least one major cloud platform (Azure, AWS, or GCP), including networking, identity (such as Entra ID/IAM), and storage services.
  • Ability to facilitate discovery workshops and translate ambiguous business needs into actionable data designs.
  • Experience delivering within enterprise architecture and governance frameworks and standards.

Nice To Haves

  • Databricks certifications, such as Data Engineer Professional, or Platform Architect accreditation.
  • Experience supporting GenAI/LLM or AI-BI use cases, such as Databricks Genie, including retrieval patterns and semantic-layer design.
  • Familiarity with data privacy regulations such as GDPR, CCPA, or HIPAA, as applicable, and audit/compliance processes.
  • Experience with infrastructure-as-code (Terraform) and CI/CD for Databricks, such as Asset Bundles and GitHub Actions/Azure DevOps.
  • Background in consulting or multi-stakeholder delivery environments.

Responsibilities

  • Lead the architecture, design, and delivery of data layers for prioritized AI use cases, from source ingestion through curated, consumption-ready data products.
  • Translate AI/BI requirements into data models, pipelines, and Databricks platform patterns, including Delta Lake, medallion architecture, and batch and streaming ingestion.
  • Define and enforce data quality, lineage, and observability standards for trusted downstream AI models and natural-language query experiences.
  • Establish reusable reference architectures and design patterns for successive use cases.
  • Co-facilitate data discovery sessions with the Lead AI/BI Engineer, business stakeholders, and data owners to identify, profile, and validate candidate data sources.
  • Assess data readiness, including completeness, quality, granularity, latency, and access, and provide findings and remediation plans.
  • Coordinate resolution of data readiness issues with source-system owners, data engineering teams, and governance stakeholders.
  • Maintain a data discovery playbook and artifacts, including source inventories, profiling results, and gap logs.
  • Architect the Databricks platform solution, including workspace topology, compute strategy, storage layout, networking, and CI/CD, aligned with enterprise standards.
  • Define development, test, and production environment strategy, promotion paths, and infrastructure-as-code practices for repeatable, auditable deployments.
  • Advise on cost optimization, performance tuning, and capacity planning across clusters, SQL warehouses, and serverless compute.
  • Track the Databricks roadmap and guide adoption decisions for Unity Catalog, Genie/AI-BI, Delta Sharing, and serverless capabilities.
  • Own end-to-end platform security design, including identity integration, workspace access, secrets management, and network isolation.
  • Design and implement the Unity Catalog governance model, including catalog and schema structure, ownership, access policies, tagging, and lineage.
  • Define and implement row-level and column-level security, dynamic data masking, and attribute-based access controls to enforce least-privilege access.
  • Own PII handling for natural-language query experiences, including classification, masking or tokenization strategies, and safeguards against sensitive-data exposure through conversational and generative interfaces.
  • Partner with enterprise security, privacy, and compliance teams to meet regulatory and audit requirements and document controls for review.
  • Serve as the primary data-architecture counterpart to the Lead AI/BI Engineer, aligning the data layer with semantic models and AI/BI experiences.
  • Provide technical direction and design reviews for data engineers delivering pipelines and data products.
  • Communicate architecture decisions, trade-offs, and risks to technical teams and business stakeholders.

Benefits

  • A culture of learning through collaboration.
  • A fun workplace culture.
  • Opportunity to work with cutting-edge technologies and contribute to company and individual success.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service