Assistant Vice President

EXL•New York, NY

About The Position

The Lead Databricks Data Engineer leads the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. The role combines hands-on data engineering with technical leadership, solution architecture, and stakeholder management, delivering scalable cloud-native data ecosystems that support analytics, AI/ML, and business intelligence. The successful candidate defines data engineering standards, mentors teams, drives modernization initiatives, and delivers high-performance data solutions. They lead Databricks and data engineering programs and work with enterprise architects, product owners, data scientists, business leaders, and security and compliance teams. The role covers cloud-native data solutions on Azure, AWS, or GCP; batch and real-time data processing; governance, lineage, cataloging, security, and data quality; and delivery planning, risk management, and resource coordination. The position is listed as full-time and located in the US.

Requirements

  • 12–15 years of overall IT experience; the posting also describes the role as requiring 10–12 years of experience.
  • At least 6 years of hands-on Databricks experience.
  • At least 7 years of Spark/PySpark-based data engineering experience.
  • Experience designing enterprise-scale Data Lake and Lakehouse solutions.
  • Strong experience leading large data transformation and modernization programs.
  • Experience processing structured, semi-structured, and unstructured data.
  • Experience supporting analytics, AI/ML, and reporting ecosystems.
  • Proven experience managing technical teams and driving architectural decisions.
  • Strong experience with one or more cloud platforms: Azure, AWS, or GCP.
  • Required technical skills include Databricks Lakehouse Platform, Databricks Workflows, Delta Lake, Unity Catalog, Structured Streaming, Databricks SQL, Delta Live Tables, and Databricks Asset Bundles.
  • Required big data skills include Apache Spark, PySpark, Spark SQL, Kafka, real-time streaming architectures, and Data Lake/Lakehouse architectures.
  • Programming skills include Python, SQL, PySpark, and Shell Scripting.
  • DevOps and automation technologies include Git, Azure DevOps, GitHub Actions, Jenkins, Terraform, Docker, and Kubernetes.
  • Data warehousing knowledge includes dimensional modeling, star and snowflake schemas, Data Vault, Master Data Management, and data governance frameworks.
  • Relevant bachelor's or master's degree in Computer Science, Engineering, Information Technology, or a related field is listed under preferred qualifications.

Nice To Haves

  • Scala.
  • Databricks Certified Data Engineer Professional certification.
  • Azure Data Engineer Associate certification.
  • AWS Data Analytics Specialty certification.
  • Google Professional Data Engineer certification.
  • TOGAF or cloud architecture certification.
  • MLOps and AI Engineering exposure.
  • Snowflake and Databricks integration experience.
  • Microsoft Fabric exposure.
  • Data Mesh implementation experience.
  • Familiarity with data governance tools such as Collibra, Alation, or Informatica.
  • Expertise in data quality frameworks such as Great Expectations or Deequ.

Responsibilities

  • Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
  • Architect and develop scalable ETL/ELT pipelines for batch and real-time processing.
  • Define enterprise data models, ingestion frameworks, and transformation strategies.
  • Lead implementation of Delta Lake, Unity Catalog, and data governance frameworks.
  • Build and optimize scalable Spark and PySpark solutions for large-volume datasets.
  • Drive cloud data platform modernization and migration initiatives.
  • Serve as technical lead for Databricks and data engineering programs.
  • Establish engineering best practices, coding standards, and architecture guidelines.
  • Conduct architecture reviews, code reviews, and solution assessments.
  • Mentor and guide Data Engineers, Technical Leads, and junior team members.
  • Collaborate with Enterprise Architects, Product Owners, Data Scientists, and business stakeholders.
  • Design cloud-native data solutions on Azure, AWS, or GCP and implement secure, scalable, resilient architectures.
  • Optimize data platform cost, performance, reliability, and availability.
  • Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
  • Define and implement monitoring, observability, and operational excellence practices.
  • Establish data governance, lineage, cataloging, and security controls; ensure compliance with organizational and regulatory requirements.
  • Implement data quality frameworks and monitoring solutions; collaborate with security and compliance teams on data protection standards.
  • Partner with business leaders to understand strategic data requirements and translate them into scalable technical solutions.
  • Drive delivery planning, risk management, and resource coordination.
  • Present architecture decisions and technical recommendations to leadership teams.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service