Experienced Databricks Database Engineer

Berkshire Hathaway Specialty Insurance • Boston, MA

114d • $125,000 - $155,000

About The Position

BHSI has an exciting opportunity for an experienced Databricks Database Engineer to join their Boston based Technology team. This position will spearhead our data engineering initiatives, focusing on designing, building, and optimizing scalable data solutions using Databricks. The ability to work both independently and within a dynamic team environment is crucial. You will mentor a team of data engineers, collaborate with cross-functional stakeholders, and define best practices to unlock the full potential of our data.

Requirements

• Minimum 5 years’ industry experience
• Proven experience (3+ years) in data engineering on Databricks, including Spark, and Delta Lake.
• Proficiency in Python, SQL, in a Microsoft Azure Environment.
• Hands-on experience with data lakes, data warehouses, and big data technologies.
• Excellent problem-solving and communication skills with the ability to interface with teammates at all levels.
• Ability to estimate work and strive to meet agreed timelines.
• Aptitude for working both individually and as part of a team.
• Prior insurance industry experience is a strong plus.

Responsibilities

• Mentor a team of data engineers in implementing Databricks-based solutions.
• Define and drive the data engineering strategy, ensuring alignment with business goals.
• Design and develop scalable data pipelines and ETL processes using Databricks.
• Optimize and tune Spark jobs for performance and efficiency.
• Recommend best practices for Databricks cluster management and data security.
• Build and maintain robust data models to support analytics and reporting needs.
• Integrate Databricks with various data sources (cloud storage, databases, APIs).
• Implement Delta Lake for reliable, scalable, and performant data lakes.
• Work closely with data analysts and stakeholders to deliver actionable insights.
• Act as the technical liaison between the data engineering team and other stakeholders.
• Stay updated on the latest Databricks and Azure features to drive continuous improvements.
• Automate repetitive processes and streamline data workflows.

Benefits

• Comprehensive Health, Dental and Vision benefits
• Disability Insurance (both short-term and long-term)
• Life Insurance (for you and your family)
• Accidental Death & Dismemberment Insurance (for you and your family)
• Flexible Spending Accounts
• Health Reimbursement Account
• Employee Assistance Program
• Retirement Savings 401(k) Plan with Company Match
• Generous holiday and Paid Time Off
• Tuition Reimbursement
• Paid Parental Leave

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level