Managing Consultant, Databricks Data Engineering Lead
About The Position
The Managing Consultant, Databricks Data Engineering Lead will lead Databricks data engineering and DataOps delivery across ingestion, transformation, orchestration, migration, integration, serving, and consumption. The role translates architecture, data models, mappings, data contracts, and migration requirements into scalable technical designs and delivery plans, and develops reusable pipelines, migration utilities, transformation frameworks, data-quality controls, reconciliation routines, and governed data products. The lead establishes engineering standards and oversees reviews, troubleshooting, technical planning, release readiness, and stabilization. The position supports batch and, where applicable, streaming processing with Databricks, Apache Spark, and Delta Lake. It coordinates delivery among technical teams and business stakeholders, mentors data engineers, promotes reusable practices, and communicates decisions, risks, dependencies, and delivery impacts to client and program leadership.
Requirements
- Bachelor’s degree from an accredited university.
- Minimum six years of experience.
- Experience in data engineering, ETL/ELT, cloud data platforms, software development, or analytics engineering.
- Hands-on experience designing and developing production Databricks solutions using Apache Spark, PySpark, Python, SQL, Delta Lake, notebooks, workflows, and jobs.
- Experience building scalable batch data pipelines and reusable ingestion, transformation, and orchestration frameworks.
- Experience with enterprise data migration, source-to-target mapping, data conversion, reconciliation, validation, and defect remediation.
- Experience implementing data-quality controls, metadata capture, lineage, logging, monitoring, and technical auditability.
- Experience with Git-based development, automated testing, CI/CD, release management, and production stabilization.
- Experience leading data engineers and delivering complex data solutions in a client-facing environment.
- Ability to obtain and maintain a Federal Public Trust.
Nice To Haves
- Databricks Certified Data Engineer Professional or comparable Databricks certification.
- Experience with Unity Catalog, Delta Live Tables, Databricks Workflows, Auto Loader, or related platform capabilities.
- Experience with cloud storage, identity and access management, secrets management, networking, and platform security.
- Experience with Azure DevOps, GitHub Actions, Jenkins, Terraform, or other infrastructure-as-code and deployment tools.
- Experience developing APIs, data services, integration patterns, or event-driven pipelines.
- Experience modernizing HR, ERP, workforce, financial, or other sensitive enterprise data.
- Experience supporting federal agencies, regulated organizations, or public-sector modernization programs.
- Experience with Agile Scrum or Kanban delivery.
Responsibilities
- Lead Databricks data engineering and DataOps delivery across ingestion, transformation, orchestration, migration, integration, serving, and consumption patterns.
- Translate architecture, data models, mappings, data contracts, and migration requirements into scalable technical designs and delivery plans.
- Design and develop reusable pipelines, migration utilities, transformation frameworks, data-quality controls, reconciliation routines, and governed data products.
- Establish standards for coding, testing, CI/CD, deployment, observability, logging, error handling, performance, security, and operational support.
- Lead engineering reviews and troubleshoot Databricks job failures, pipeline bottlenecks, data-quality issues, and performance problems.
- Support batch and, where applicable, streaming data-processing patterns using Databricks, Apache Spark, and Delta Lake.
- Coordinate technical delivery with data architects, source-system teams, integration engineers, security specialists, governance leads, testing teams, and business stakeholders.
- Lead technical estimates, capacity planning, backlog refinement, sprint execution, release readiness, and stabilization activities.
- Mentor data engineers and ensure reusable code, design patterns, runbooks, and lessons learned are adopted across later modernization waves.
- Communicate technical decisions, risks, dependencies, and delivery impacts to client and program leadership.
Benefits
- Medical, prescription, dental, and vision insurance
- Personal and family sick time and company-paid holidays
- May be eligible for a discretionary variable incentive bonus
- Parental leave and adoption assistance
- 401(k) retirement plan
- Basic life and supplemental life insurance
- Health savings account
- Dental/vision and dependent care flexible spending accounts
- Short-term and long-term disability
- Student loan paydown
- Tuition reimbursement
- Personal development and learning opportunities
- Skills development and certifications
- Employee referral program
- Corporate-sponsored events and community outreach
- Emergency back-up childcare program
- Mobility stipend
- Flexible benefits package
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
