Senior Databricks & AI Architect/Lead - Only W2 - (JK)

Central Business Solutions • Dallas, NY

7d

About The Position

We are seeking a Senior Databricks & AI Architect to lead the evolution of the modern data platform and AI roadmap. The role is responsible for designing scalable Spark-based architectures and leading data architecture, AI workflow design, data engineering standards, and governance for insurance data and documents. The position is a long-term contract and requires an initial onsite presence in Dallas, TX, New Jersey, New York, or California, with the possibility of converting to hybrid later.

Requirements

• 8+ years of experience in data engineering or architecture.
• At least 3 years of hands-on leadership within the Databricks ecosystem.
• Expert-level knowledge of Spark, including PySpark or Scala, Delta Lake, and Databricks SQL.
• Proven experience building and deploying AI solutions.
• Deep understanding of Agentic AI, including autonomous, goal-oriented agents beyond simple chatbots.
• Strong background in data modeling and creating reusable ETL/ELT templates and frameworks.

Responsibilities

• Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform.
• Lead the design of AI workflows, including multi-agent Agentic AI systems, LLM orchestration with LangChain or LlamaIndex, and RAG patterns for insurance policy and regulatory documents.
• Standardize data engineering templates using Medallion architecture.
• Implement data modeling approaches, including Star/Snowflake and Vault 2.0, for structured and unstructured insurance data.
• Ensure designs comply with industry regulations using Databricks Unity Catalog.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed