Senior Databricks Solutions Architect

Prodapt • Irving, TX

6d • Hybrid

About The Position

Prodapt is seeking a Senior Databricks Solution Architect to lead the architecture and technical execution of a migration from Palantir Foundry to Databricks. The role includes assessing the existing Palantir application, defining the target Databricks architecture, mapping current capabilities to Databricks services, and providing technical leadership throughout the migration. The solution encompasses data ingestion and transformation, AI-based information extraction from structured and unstructured documents, large-scale data comparison, dashboards and drill-down reporting, chatbot capabilities, and business workflows and applications.

The successful candidate will bring strong expertise in Databricks, Lakehouse architecture, data engineering, AI/ML workloads, and cloud-native data platforms. They will decompose existing Palantir Foundry capabilities into equivalent or improved Databricks-based solutions, guide engineering and AI teams, and collaborate with business stakeholders, Palantir SMEs, cloud architects, and application teams. This is a hybrid role in Dallas, Texas.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related discipline.
• 14+ years of experience in data engineering or data architecture.
• 5+ years of experience working with modern cloud data platforms.
• Strong hands-on Databricks experience.
• Proven experience designing enterprise-scale data platforms.
• Experience leading technical teams and architecture initiatives.
• Strong communication and stakeholder-management skills.
• Ability to translate business requirements into scalable technical architectures.

Responsibilities

• Define the target-state Databricks architecture and design a scalable, secure Lakehouse architecture, including Bronze, Silver, and Gold layers.
• Design data ingestion, transformation, processing, and serving patterns, including batch and near-real-time processing.
• Define standards for Delta Lake, Unity Catalog, compute, storage, workloads, and environments.
• Analyze the existing Palantir Foundry application, including pipelines, datasets, applications, workflows, and AI capabilities.
• Map Palantir capabilities to Databricks capabilities, identify what can migrate directly versus what requires redesign, and define migration patterns.
• Develop migration roadmaps, architecture decisions, and technical dependencies, and identify opportunities to improve scalability, maintainability, and performance.
• Design data pipelines using Lakeflow / Spark Declarative Pipelines, Apache Spark, and Delta Lake.
• Design ingestion from databases, APIs, files, documents, and other enterprise data sources.
• Establish data quality, validation, lineage, and observability mechanisms, and design incremental processing and change-data-capture patterns where required.
• Optimize Delta tables, Spark workloads, and Databricks compute for performance and cost.
• Architect document information extraction, RAG and LLM-based solutions where applicable, and evaluate Databricks capabilities for document processing, embeddings, vector search, LLM inference, model serving, and AI agents or conversational applications.
• Define approaches for migrating existing Palantir AIP-based extraction capabilities into Databricks and ensure AI solutions are scalable, secure, explainable, and production-ready.
• Design target architectures for dashboards, drill-down reporting, analytical applications, semantic and data-serving layers, chatbots, conversational analytics, and integration with downstream applications and APIs.
• Implement enterprise data governance using Unity Catalog; define catalog, schema, table, volume, access-control, RBAC, lineage, auditing, and governance strategies.
• Ensure compliance with enterprise security and privacy requirements.
• Lead data engineers, ML/AI engineers, and developers; conduct architecture reviews and technical design sessions.
• Create HLD/LLD documents, architecture diagrams, and migration plans; establish Databricks development standards and best practices.
• Review code and implementation approaches, collaborate with stakeholders and technical teams, and troubleshoot complex architectural and performance issues.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior