Lead Digital Data Engineer (Databricks)

BJ's Wholesale Club • BJ's Club Support Center Marlborough, MA

10d • Hybrid

About The Position

The Lead Digital Data Engineer owns the design, development, and evolution of BJ’s Databricks Lakehouse. Working with product managers, program managers, and business unit heads, this role turns digital, fulfillment, and operational data into trusted, governed, self-service data products and leads adoption of AI-powered analytics, including Genie and Databricks agents. The position is hybrid, with in-office work Tuesday through Thursday at BJ’s Club Support Center in Marlborough, Massachusetts, and remote work Monday and Friday.

The role also owns and prioritizes the digital data and analytics roadmap with product leaders, and leads and mentors a distributed team of up to 10 engineers across onshore and offshore locations. It drives Agile delivery and provides regular status and risk updates to senior leadership. The stated pay range is $128,500.00–$164,000.00.

Requirements

• Bachelor's degree in Computer Science, Information Technology, or equivalent.
• At least 7 years of experience in data engineering, including at least 3 years of hands-on Databricks experience in production.
• Expert-level Apache Spark (PySpark and SQL), Delta Lake, and Lakehouse architecture experience.
• Experience with Unity Catalog, Databricks Workflows, and SQL Warehouses.
• Experience integrating AWS services, including S3, Kafka (MSK/Confluent), DynamoDB, Lambda, and IAM.
• Experience building data-quality and validation frameworks.
• Experience with Genie Spaces, AI/BI dashboards, and building Databricks agents or GenAI solutions using Mosaic AI and MLflow.
• Experience building large-scale distributed data systems on AWS, Azure, or GCP.
• Experience working in an Agile Scrum environment and leading distributed teams.
• Experience with SDLC, CI/CD, version control, and test automation for data pipelines.

Responsibilities

• Design and maintain a scalable Databricks Lakehouse using Delta Lake and the Bronze/Silver/Gold medallion architecture as the analytics single source of truth following migration from AWS Redshift.
• Build batch and real-time data pipelines integrating Kafka, DynamoDB, S3, and other AWS services with Databricks.
• Ingest and integrate Adobe Analytics, AEP, and Google Analytics data alongside transactional and operational data.
• Implement data-quality validation frameworks with automated checks, alerting, and SLAs.
• Lead data governance through Unity Catalog, including access control, lineage, and classification; monitor pipeline health, freshness, and cost.
• Design reusable Genie Spaces and AI/BI dashboards with certified metrics for business self-service.
• Develop and productionize Databricks agents and AI use cases using Mosaic AI, MLflow, Model Serving, and Vector Search.
• Optimize performance and cost across clusters, SQL Warehouses, and Delta tables.
• Establish CI/CD for data pipelines using Databricks Asset Bundles, Git workflows, automated testing, and Terraform.
• Own and prioritize the digital data and analytics roadmap with product leaders.
• Lead and mentor a distributed team of up to 10 onshore and offshore engineers, drive Agile delivery, and provide regular status and risk updates to senior leadership.

Benefits

• Weekly pay.
• Daily Pay access to earned pay before payday; a fee may apply.
• Complimentary The Club Card Membership and a free Supplemental Membership for someone in the household.
• Paid time off, including vacation, personal, sick days, holidays, bereavement, and jury duty leave.
• Three medical plan options.
• Optional dental and vision coverage.
• Health Savings Account (HSA) and flexible spending account options.
• 401(k) retirement savings plan with company match, available to team members 18 and older.
• Employee Stock Purchase Plan, allowing eligible team members to purchase BJ’s common stock at a 15% discount.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior