Data Engineer

Brown & Brown Insurance•Plano, TX

15h

About The Position

Brown & Brown is seeking a Data Engineer to join its growing team in Plano, Texas. The role advances the company’s Data & AI Platform capabilities by designing, building, and operating modern data solutions for AI/ML, analytics, business intelligence, and enterprise insights. The engineer works with a collaborative engineering team and partners with engineers, product owners, architects, and business stakeholders to translate requirements into production-ready data pipelines and data products. This is a hands-on role spanning the full development lifecycle: solution design, development, testing, deployment, monitoring, troubleshooting, and ongoing optimization. The engineer builds scalable and governed batch and streaming solutions using Databricks, Delta Lake, Python, PySpark, Apache Spark, and SQL. The role also supports data governance, quality, observability, workflow orchestration, production operations, and the delivery of data assets for analytics and AI use cases.

Requirements

Bachelor’s degree in computer science, engineering, mathematics, information systems, or a related STEM field, or equivalent education and professional experience.
At least 4 years of experience in data engineering, software engineering, or a related technical discipline, including developing and supporting production data solutions.
Hands-on experience with Apache Spark/PySpark, Delta Lake, Python, and SQL.
Experience building and supporting production batch and/or streaming data pipelines.
Experience with Databricks capabilities including Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, Delta Lake, and Unity Catalog.
Experience developing ETL/ELT pipelines to ingest, transform, validate, and publish enterprise data.
Experience with workflow orchestration, including scheduling, dependencies, parameters, retries, notifications, and monitoring.
Understanding of Unity Catalog and governance concepts, including catalogs, schemas, tables, views, lineage, access controls, and permissions.
Experience with Git and CI/CD practices, and exposure to Infrastructure as Code technologies such as Terraform or Databricks Asset Bundles.
Strong Python and SQL development skills and ability to create maintainable, testable, production-ready data solutions.
Experience troubleshooting and optimizing Spark workloads using Spark UI, job logs, query plans, partitioning, and join optimization.
Understanding of relational, dimensional, logical, and physical data modeling concepts.
Experience across the engineering lifecycle, from requirements and development through testing, deployment, monitoring, and production support.
Strong problem-solving and analytical skills, including diagnosing and resolving data and production issues.
Strong written and verbal communication skills and ability to collaborate with technical and business teams.
Experience working within Agile delivery models such as Scrum, Kanban, or SAFe.
Advanced Python and PySpark experience for data processing, automation, and reusable engineering components.
A strong engineering mindset emphasizing quality, automation, observability, scalability, and continuous improvement.

Nice To Haves

Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
Experience with cloud storage technologies such as Azure Data Lake Storage Gen2, Amazon S3, OneLake, or Google Cloud Storage.
Experience with modern data technologies such as Snowflake, Apache Iceberg, Apache Hudi, or BigQuery.
Experience building ingestion pipelines using Databricks Auto Loader.
Experience developing streaming solutions using Structured Streaming.
Experience designing or consuming REST APIs and web services.
Experience implementing data-quality, metadata, governance, or lineage capabilities.
Experience developing data products for AI, generative AI, machine learning, semantic models, APIs, advanced analytics, Genie Spaces, or AI agents.
Experience working with insurance and health-domain data.
Practical experience using AI-assisted development tools such as GitHub Copilot, Claude, or Databricks AI-assisted development capabilities.
Strong experience building and supporting production-scale Databricks Lakehouse solutions.
Strong Python skills applied to reusable data engineering frameworks, automation, libraries, and utilities.
Experience with Delta Lake and modern lakehouse architectures.
Experience developing governed datasets and data products for analytics, AI, and generative AI use cases.
Experience with automated testing, deployment pipelines, Infrastructure as Code, and modern software engineering practices.
Demonstrated ability to troubleshoot complex data problems and improve existing solutions’ reliability, performance, and maintainability.

Responsibilities

Design, develop, test, deploy, and support scalable batch and streaming data solutions.
Build Databricks-native solutions using Delta Lake, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
Develop production-grade ingestion, transformation, and processing pipelines using Python, PySpark, Apache Spark, and SQL.
Build reliable, governed pipelines that improve enterprise data accuracy, consistency, accessibility, and trustworthiness.
Develop reusable components, libraries, templates, and automation to improve engineering productivity and consistency.
Implement data-quality validation, monitoring, observability, and remediation processes for critical data assets.
Integrate governance capabilities, including metadata, lineage, classification, certification, ownership, and access controls.
Develop and maintain governed data assets supporting AI, machine learning, analytics, semantic models, APIs, Genie Spaces, and AI agents.
Build and maintain workflow orchestration, including scheduling, dependencies, parameters, retries, notifications, and operational monitoring.
Troubleshoot and optimize workloads using Spark diagnostics, job logs, query plans, partitioning strategies, join optimization, and Adaptive Query Execution.
Support production solutions through monitoring, incident resolution, performance tuning, migrations, upgrades, and cost optimization.
Participate in technical design discussions, code reviews, testing, and continuous improvement of engineering practices.
Collaborate with product owners, architects, operations teams, and engineers to resolve dependencies and deliver solutions.
Maintain technical documentation for pipelines, data models, operational procedures, and engineering solutions.