Lead Data Engineer (AWS)

State Farm • Dunwoody, GA

4d • Hybrid

About The Position

State Farm's Innovation Group is seeking a Data Engineer or Lead Data Engineer to join its product execution team within SAIL (State Farm Applied Innovation Lab), focused on Agency and Digital. The role partners with software engineers, product teams, and business partners to design and build data capabilities supporting experimentation, product development, and emerging technology solutions.

The team works in environments where the problem, data, and solution may not be fully defined at the outset. Engineers are expected to navigate ambiguity, learn new technologies and domains, and turn complex data needs into practical, scalable solutions. The work includes writing code to move and transform data, collaborating to determine how data can be accessed and used, and designing and maintaining platforms for structured and unstructured data.

The role calls for production-quality data engineering across the solution lifecycle, including design, development, deployment, monitoring, and maintenance. Depending on the initiative, it may also support data science, machine learning, and generative AI efforts, from exploration and experimentation through engineering, deployment, and ongoing operation. At the Lead Data Engineer level, the role includes technical leadership, mentoring engineers, influencing architecture and engineering practices, and helping shape the team's data capabilities and solutions.

This is a hybrid position. Qualified candidates must live or relocate within 180 miles of a listed hub: Bloomington, Illinois; Richardson, Texas; Tempe, Arizona; or Dunwoody, Georgia. The work arrangement includes time working from home and in the office. Applicants must be eligible to work lawfully in the U.S. immediately; State Farm will not sponsor U.S. work authorization for this opportunity.

Requirements

• At least 3 years of Python development for extracting, transforming, processing, and moving data.
• Proficiency writing SQL queries and working with relational databases.
• Experience designing, building, and maintaining ETL/ELT data pipelines.
• Experience working with structured, semi-structured, and unstructured data, including CSV, JSON, and text formats.
• Experience with public cloud platforms, preferably AWS, including services such as S3, Glue, and Lambda.
• Experience using Git and modern version-control practices.
• Familiarity with CI/CD practices and deploying code into shared or production environments.
• Comfort working in Linux and command-line environments.
• Understanding of software engineering principles, including testing, maintainability, observability, and reusable design.
• Ability to troubleshoot data quality, pipeline, integration, and performance issues.
• Clear communication and effective collaboration with technical and non-technical teams.
• Ability to understand the business problem behind a request and adapt as technologies and needs evolve.
• Qualified candidates must live or relocate within 180 miles of a listed hub location.
• Must be eligible to lawfully work in the U.S. immediately; employer sponsorship for U.S. work authorization is not available.

Nice To Haves

• Experience designing data architectures for new products, experiments, or emerging technology use cases.
• Familiarity with machine learning concepts and the end-to-end ML lifecycle, including data preparation, feature engineering, model development, evaluation, deployment, monitoring, and iteration.
• Experience partnering with data scientists or ML engineers to operationalize models and build supporting data pipelines and infrastructure.
• Familiarity with generative AI, large language models (LLMs), and data considerations for AI-enabled applications.
• Experience designing or building LLM-based workflows, including agentic workflows using tools, APIs, retrieval, orchestration, or multi-step reasoning.
• Experience using AI-assisted development tools to accelerate coding, testing, debugging, documentation, and technical exploration.
• Ability to critically evaluate AI-generated code and outputs and apply appropriate engineering, security, testing, and governance practices.
• Experience with APIs, event-driven architectures, or streaming data.
• Familiarity with object storage and modern data platforms.
• Experience working with large or complex datasets.
• Understanding of data governance, security, privacy, lineage, and access-management considerations.
• Experience working in Agile or product-oriented engineering environments.
• Ability to translate business needs into technical data requirements and solutions.
• Experience working across teams and managing technical dependencies.
• Comfort experimenting with new technologies and solving problems where the path forward is not fully defined.

Responsibilities

• Write code to extract, move, transform, and process data.
• Partner with software engineers, product teams, and business partners to determine how data can be accessed, analyzed, and used.
• Design, build, and maintain platforms for structured and unstructured data.
• Design, build, deploy, monitor, and maintain production-quality data solutions.
• Depending on the initiative, contribute to data science, machine learning, and generative AI efforts across exploration, experimentation, engineering, deployment, and ongoing operation.
• At the Lead Data Engineer level, provide technical leadership and mentor engineers.
• At the Lead Data Engineer level, influence architecture and engineering practices and help shape the team's data capabilities and solutions.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed