Managing Consultant, Databricks Data Engineering Lead
About The Position
The Managing Consultant, Databricks Data Engineering Lead, leads Databricks data engineering and DataOps delivery across data ingestion, transformation, orchestration, migration, integration, serving, and consumption. The role translates architecture, data models, mappings, data contracts, and migration requirements into scalable technical designs and delivery plans, and develops reusable pipelines, migration utilities, transformation frameworks, data-quality controls, reconciliation routines, and governed data products. The position establishes engineering standards for coding, testing, CI/CD, deployment, observability, logging, error handling, performance, security, and operational support. It leads engineering reviews and troubleshooting, supports batch and applicable streaming patterns, and coordinates delivery with architecture, source-system, integration, security, governance, testing, and business teams. The role also leads estimates, capacity planning, backlog refinement, sprint execution, release readiness, and stabilization; mentors data engineers; and communicates technical decisions, risks, dependencies, and delivery impacts to client and program leadership.
Requirements
- Bachelor’s degree from an accredited university.
- Minimum of six or more years of experience.
- Experience in data engineering, ETL/ELT, cloud data platforms, software development, or analytics engineering.
- Hands-on experience designing and developing production Databricks solutions using Apache Spark, PySpark, Python, SQL, Delta Lake, notebooks, workflows, and jobs.
- Experience building scalable batch data pipelines and reusable ingestion, transformation, and orchestration frameworks.
- Experience with enterprise data migration, source-to-target mapping, data conversion, reconciliation, validation, and defect remediation.
- Experience implementing data-quality controls, metadata capture, lineage, logging, monitoring, and technical auditability.
- Experience with Git-based development, automated testing, CI/CD, release management, and production stabilization.
- Experience leading data engineers and delivering complex data solutions in a client-facing environment.
- Ability to obtain and maintain a Federal Public Trust.
Nice To Haves
- Databricks Certified Data Engineer Professional or comparable Databricks certification.
- Experience with Unity Catalog, Delta Live Tables, Databricks Workflows, Auto Loader, or related platform capabilities.
- Experience with cloud storage, identity and access management, secrets management, networking, and platform security.
- Experience with Azure DevOps, GitHub Actions, Jenkins, Terraform, or other infrastructure-as-code and deployment tools.
- Experience developing APIs, data services, integration patterns, or event-driven pipelines.
- Experience modernizing HR, ERP, workforce, financial, or other sensitive enterprise data.
- Experience supporting federal agencies, regulated organizations, or public-sector modernization programs.
- Experience with Agile Scrum or Kanban delivery.
Responsibilities
- Lead Databricks data engineering and DataOps delivery across ingestion, transformation, orchestration, migration, integration, serving, and consumption patterns.
- Translate architecture, data models, mappings, data contracts, and migration requirements into scalable technical designs and delivery plans.
- Design and develop reusable pipelines, migration utilities, transformation frameworks, data-quality controls, reconciliation routines, and governed data products.
- Establish standards for coding, testing, CI/CD, deployment, observability, logging, error handling, performance, security, and operational support.
- Lead engineering reviews and troubleshoot Databricks job failures, pipeline bottlenecks, data-quality issues, and performance problems.
- Support batch and, where applicable, streaming data-processing patterns using Databricks, Apache Spark, and Delta Lake.
- Coordinate technical delivery with data architects, source-system teams, integration engineers, security specialists, governance leads, testing teams, and business stakeholders.
- Lead technical estimates, capacity planning, backlog refinement, sprint execution, release readiness, and stabilization activities.
- Mentor data engineers and ensure reusable code, design patterns, runbooks, and lessons learned are adopted across later modernization waves.
- Communicate technical decisions, risks, dependencies, and delivery impacts to client and program leadership.
Benefits
- Medical, prescription, dental, and vision insurance
- Personal and family sick time
- Company-paid holidays
- Potential eligibility for a discretionary variable incentive bonus
- Parental leave
- Adoption assistance
- 401(k) retirement plan
- Basic life and supplemental life insurance
- Health Savings Account
- Dental/vision and dependent care Flexible Spending Accounts
- Short-term and long-term disability
- Student loan paydown
- Tuition reimbursement
- Personal development and learning opportunities
- Skills development and certifications
- Employee referral program
- Corporate-sponsored events and community outreach
- Emergency back-up childcare program
- Mobility stipend
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Manager
