Principal Data Engineer
About The Position
The Life, Annuity, and Long-Term Care Data Solutions team is seeking a Principal Data Engineer with strong Databricks expertise and proven experience in a Tech Lead capacity. This role involves leading and mentoring data engineering tech leads across multiple build teams that use DataBricks and a data engineering CoE (Center of Excellence). The position guides teams through the full software development lifecycle, defines best practices, establishes common code frameworks, elevates engineering maturity, evaluates new DataBricks features, and ensures consistent delivery of high-quality data products and analytical solutions. The team operates within a high-performing, mission-driven organization of 25 individuals across four agile teams, emphasizing collaboration with solution architects, agile team tech leads, data product owners, and partners in security, cloud, testing, and data modeling. The platform utilizes a modern streaming architecture supporting transactional and analytical pipelines, based on Medallion and OneLake design principles. A key outcome is overseeing the implementation of DataBricks to replace legacy analytical platforms, driving new and better business outcomes.
Requirements
- 3–5 years of hands-on Databricks experience with other complementary technologies including data governance, reporting and dashboard, and source system technologies.
- Proven experience in a Tech Lead role, including leading developers and distributing/decomposing work.
- Strong communication skills with the ability to mentor, influence, and align cross-functional stakeholders.
- Ability to engage proactively with business leaders to recommend solutions aligned to strategy and support data self-service capabilities.
- Solid grounding in general analytics, data warehousing, and modern data engineering practices.
- Experience working in Agile frameworks and writing high-quality user stories.
- Ability to navigate ambiguity and provide clarity for your team and partners.
Nice To Haves
- Azure and Fabric cloud experience
- Insurance industry business process and data knowledge.
- Medical data experience.
- Familiarity with data governance and data quality needs for business consumers and how they integrate with pipeline design.
Responsibilities
- Leading 3–4 build teams, providing technical direction, coaching, mentoring, and technical governance.
- Directly leading some data engineers for POCs, building coding frameworks that will be shared across teams, and other re-usable data assets.
- Performing hands-on designing, building, testing, and supporting analytical pipelines.
- Establishing and enforcing design, engineering, and coding standards, including code reviews.
- Leading CI/CD and designing reusable data tests for future releases.
- Expanding expertise into AI, ML, Lakebase, and other capabilities available in the DataBricks platform to ensure these features are implemented as needed.
- Driving Spark performance optimization and leading platform cost optimization opportunities.
- Managing production monitoring, troubleshooting, and defect resolution.
Benefits
- Competitive Compensation & Total Rewards Incentives
- Comprehensive Healthcare Coverage
- Multiple 401(k) Savings Plan Options
- Auto Enrollment in Employer-Directed Retirement Account Feature (100% employer-funded!)
- Generous Paid Time Off – Including 12 Paid Holidays, Volunteer Time Off and Paid Family Leave
- Disability, Life, and Long Term Care Insurance
- Tuition Reimbursement, Student Loan Repayment and Training & Certification Support
- Wellness support including gym membership reimbursement and Employee Assistance Program resources (work/life support, financial & legal management)
- Caregiver and Mental Health Support Services
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Principal
Education Level
No Education Listed
