Lead Data Engineer

Brown & Brown Insurance • Plano, TX

18h

About The Position

Brown & Brown is seeking a Lead Data Engineer to join its growing team in Plano, Texas. The role advances the company’s Data & AI platform by designing, building, and operating modern data solutions that support AI, machine learning, analytics, business intelligence, and enterprise insights. It combines hands-on engineering with technical leadership and partners with product owners, architects, business stakeholders, and engineering teams to turn business needs into practical, production-ready solutions.

The Lead Data Engineer owns data solutions across the engineering lifecycle, from architecture and design through development, deployment, and ongoing operations. The role establishes engineering standards and reusable patterns, improves platform capabilities, mentors engineers, resolves complex technical challenges, and helps deliver reliable, governed, scalable, and cost-effective data products.

Requirements

• Bachelor’s degree in computer science, engineering, mathematics, information systems, or a related STEM field, or equivalent education and professional experience.
• 8+ years of experience in data engineering or related software/data engineering disciplines, including significant hands-on experience building and supporting production data solutions.
• Strong hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Python, and SQL.
• Experience designing and operating production-grade batch and streaming data pipelines.
• Experience with Databricks Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, Delta Lake, and Unity Catalog.
• Experience designing reliable workflow orchestration, including scheduling, dependencies, parameters, retries, notifications, and operational monitoring.
• Experience implementing data governance and access controls with Unity Catalog, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions.
• Experience with Terraform, Databricks Asset Bundles, or comparable Infrastructure as Code approaches, along with Git and CI/CD practices.
• Strong experience troubleshooting and optimizing Spark workloads using tools and techniques such as Spark UI, job logs, query plans, partitioning, join optimization, and Adaptive Query Execution.
• Strong understanding of conceptual, logical, relational, dimensional, and physical data modeling.
• Experience delivering ETL/ELT solutions throughout the engineering lifecycle, from requirements and design through deployment and production support.
• Ability to balance engineering quality, delivery speed, scalability, resiliency, and cost while making pragmatic technical trade-offs.
• Strong problem-solving skills and demonstrated ability to diagnose and resolve complex data and production issues.
• Demonstrated technical leadership, including solution design, code reviews, engineering standards, mentoring, and unblocking teams.
• Ability to influence technical direction, facilitate alignment, and build consensus across engineering teams and stakeholders without relying on formal authority.
• Experience working within Agile delivery models such as Scrum, Kanban, or SAFe.

Nice To Haves

• Prior experience as a technical lead, lead engineer, or senior engineer with responsibility for technical design and engineering standards.
• Advanced Python and/or Scala development experience.
• Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
• Experience with cloud storage technologies such as Azure Data Lake Storage Gen2, Amazon S3, OneLake, or Google Cloud Storage.
• Experience with modern data platforms and open table technologies such as Snowflake, Apache Iceberg, Apache Hudi, or BigQuery.
• Experience with messaging and event-streaming technologies.
• Experience with NoSQL technologies such as Cassandra or HBase.
• Experience designing or consuming REST APIs and web services.
• Experience developing governed data products supporting AI, Generative AI, machine learning, semantic models, APIs, advanced analytics, Genie Spaces, or AI agents.
• Experience implementing enterprise-scale data governance, data quality, metadata, and lineage capabilities.
• Experience working with insurance or health data.
• Practical experience using AI-assisted development tools such as GitHub Copilot, Claude, or Databricks AI-assisted development capabilities to improve engineering productivity.
• Deep experience designing, building, and optimizing production-scale Databricks Lakehouse solutions.
• Experience building high-volume or near-real-time ingestion solutions.
• Advanced Python experience developing reusable engineering frameworks, libraries, automation, and platform capabilities.
• Experience building governed, reusable data products for AI and Generative AI applications.
• Demonstrated ability to simplify complex technical architectures and establish reusable engineering patterns across teams.
• A track record of raising engineering standards while helping teams deliver incrementally and predictably.
• Strong technical judgment and ability to balance speed, simplicity, scalability, reliability, governance, and cost.

Responsibilities

• Lead design, development, deployment, and operational support of scalable batch and streaming data solutions.
• Design and build Databricks-native data engineering solutions using Delta Lake, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
• Develop production-grade ingestion, transformation, and processing pipelines using Python, PySpark, Apache Spark, and SQL.
• Define technical designs, engineering standards, and reusable patterns that improve scalability, reliability, maintainability, and developer productivity.
• Build and maintain governed data solutions that improve enterprise data accuracy, consistency, accessibility, and trustworthiness.
• Develop reusable frameworks, libraries, templates, and automation for data ingestion, transformation, feature engineering, and consumption.
• Implement data-quality validation, monitoring, observability, and remediation for critical data assets.
• Integrate solutions with enterprise governance capabilities, including metadata, lineage, classification, certification, ownership, and access controls.
• Develop governed data assets supporting AI, machine learning, advanced analytics, semantic models, APIs, Genie Spaces, and AI agents.
• Troubleshoot and optimize production workloads using Spark diagnostics, job logs, query plans, partitioning strategies, join optimization, and Adaptive Query Execution.
• Operate and continuously improve data solutions through performance monitoring, incident resolution, migrations, upgrades, observability, and cost optimization.
• Implement automated infrastructure and environment-aware deployments using Infrastructure as Code, Git, and CI/CD practices.
• Partner with product owners, architects, operations teams, and engineering teams to prioritize capabilities, address technical dependencies, and expand platform adoption.
• Lead technical design discussions, conduct code and design reviews, drive architectural alignment, and proactively identify and remove technical blockers.
• Mentor engineers and foster engineering excellence, collaboration, continuous learning, and accountability.

Benefits

• Medical/Rx, dental, vision, life insurance, and disability insurance.
• Employee Stock Purchase Plan (ESPP).
• 401(k).
• Student loan assistance.
• Tuition reimbursement.
• Free mental health and enhanced advocacy services.
• Paid time off and holidays.
• Preferred partner discounts.
• Additional benefits may be available; eligibility criteria or waiting periods may apply, and offerings may vary by subsidiary or geographic location.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior