Data Manager (Databricks)

Allata•Dallas, TX

49d

About The Position

Allata is a global consulting and technology services firm founded in 2016, with more than 350 employees across the US, India, and Argentina. It partners with large enterprises to develop AI, data foundations, and governance frameworks. The role works on agile, cross-functional client teams focused on technology modernization, data value, and digital experiences. The Databricks Manager leads client-facing enterprise data platform implementations, combining architecture ownership with technical leadership. The role defines platform architecture, security, scalability, cost management, and operating models, while guiding the design, development, deployment, and optimization of data pipelines and products on Lakehouse and enterprise data warehouse platforms, with an emphasis on Databricks. The manager owns the full lifecycle of data products and standardizes patterns, tools, and practices across large-scale, vertical-specific client implementations. The position also serves as a technical authority and mentor for data engineering teams, collaborating with business and technology stakeholders to translate requirements into technical solutions and roadmaps. It develops and deploys processing modules and pipelines, applies DevOps practices, and resolves complex data quality, performance, and availability issues.

Requirements

Previous experience in an architect or lead technical role on enterprise data platforms.
Hands-on experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming.
Strong expertise in Spark (PySpark and/or Scala), Spark SQL, and distributed job optimization.
Background in data warehouse and Lakehouse design and practical familiarity with Medallion, Lambda, and Kappa patterns.
Experience integrating SaaS, ETL, or connector tools such as Fivetran; orchestration platforms such as Airflow, Azure Data Factory, or Data Fabric; and ELT/ETL tooling.
Experience with relational and hybrid databases such as MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora, or equivalents.
Proficiency in CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar tools, and packaging or deploying artifacts such as .whl files and containers.
Experience with batch and streaming processing, file compaction, partitioning strategies, and storage tuning.
Good understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking.
Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, native cloud monitoring, or equivalents.
Ability to identify, troubleshoot, and resolve complex data issues.
Strong teamwork and communication skills, intellectual curiosity, and ability to work collaboratively with cross-functional teams.
Commitment to delivering high-quality, accurate, and reliable data product solutions.
Willingness to embrace new tools, technologies, and methodologies.
Innovative thinking and a proactive approach to overcoming challenges.

Nice To Haves

Automation experience supporting deployment and integration workflows, including trunk-based development using Azure DevOps, Jenkins, Octopus, or similar automation services.
Advanced proficiency in PySpark for advanced data processing tasks.
Advanced proficiency in Spark workflow optimization and orchestration using tools such as Asset Bundles or DAG orchestration.
Databricks Certified Data Engineer or Databricks Certified Professional Architect certification.
Cloud architect or data certifications in AWS, Azure, or GCP.

Responsibilities

Define data platform architecture for Lakehouse and EDW environments, including reference patterns, technology selection, and integration blueprints.
Design conceptual, logical, and physical data models for multi-tenant and vertical-specific data products, and standardize data layers.
Establish data governance, metadata, cataloging, lineage, data contracts, and classification practices.
Define security and compliance controls, including access management, masking, encryption, network segmentation, and auditing.
Architect scalability, high availability, disaster recovery, capacity, and cost-management strategies for cloud and hybrid deployments.
Lead selection and integration of platform components, including Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory or Data Fabric, orchestration, and monitoring tools.
Design and enforce CI/CD patterns for data artifacts, including testing, automated deployments, and rollback strategies.
Define batch and streaming ingestion patterns, compaction approaches, partitioning, and storage layouts to optimize input/output and costs.
Specify observability practices, including metrics, SLAs, dashboards, structured logging, tracing, and alerting.
Provide technical leadership and mentoring to data engineering teams and conduct architecture and code reviews for critical components.
Collaborate with data product owners and security, infrastructure, BI, and ML stakeholders to translate requirements into technical solutions and roadmaps.
Design, develop, test, and deploy processing modules using Spark, Spark SQL, and database stored procedures where applicable.
Build and optimize data pipelines on Databricks and complementary database engines.
Implement DevOps practices, including infrastructure as code, CI/CD, automated testing, and version control.
Troubleshoot complex data quality, performance, and availability issues and implement continuous improvements.