Senior Data Engineer (Databricks)

Gulf Coast Automation Group•Chicago, IL
•Hybrid

About The Position

TalentFish is hiring a Senior Data Engineer for a full-time, direct-hire position in Chicago, Illinois. The role is part of a multi-year effort to transform the client’s Data & AI function and build a modern data platform. The engineer will be a core hands-on contributor building a new Azure Databricks platform; Databricks is established, but production pipelines have not yet been created. The role is an individual-contributor position rather than a people-management, architecture, AI/LLM, or BI/reporting role. The platform architecture is defined, and the engineer will implement it and help raise engineering standards as the team grows. The engineer will design and deliver pipelines, transformations, and models, owning work from source ingestion through validation, transformation, governance, and outputs for downstream analytics, BI, and AI/ML. The work includes migrating legacy data warehouses into a centralized Databricks lakehouse, establishing engineering practices, optimizing platform performance, and collaborating across technical and business teams. The role has executive sponsorship and enterprise-wide visibility. The work arrangement is hybrid, with at least three days onsite and two days remote each week. The interview process includes a live assessment of Python, SQL, and practical data-pipeline logic.

Requirements

  • Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.
  • At least 6 years of hands-on data engineering experience, including end-to-end ownership of pipelines covering ingestion, validation, transformation, and governance.
  • Hands-on Databricks experience, with Azure preferred; AWS or GCP experience is acceptable. Experience should include Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, and Workflows/Jobs.
  • Strong Python, SQL, and Apache Spark coding skills beyond basic scripting.
  • Experience designing and building scalable ETL/ELT pipelines, including reusable, metadata-driven pipelines.
  • Experience with object storage, lakehouse engines, and orchestration tools, as well as streaming or CDC pipelines.
  • Data modeling, schema design, and performance-tuning experience.
  • Working knowledge of Git-based workflows, CI/CD, test automation, monitoring, and alerting.
  • Strong troubleshooting skills and understanding of scalability, reliability, and maintainability.
  • Ability to explain what was built, why it mattered, and how it addressed a business or platform problem.
  • Authorization to work in the U.S. without current or future visa sponsorship.

Nice To Haves

  • Experience building or migrating a data platform from the ground up.
  • Infrastructure as Code experience with Terraform, Bicep, or ARM, and Governance as Code experience.
  • Multi-cloud background with depth in Azure.
  • Legacy data warehousing and/or Hadoop experience.
  • Deep Databricks experience beyond basic pipelines, such as enterprise-scale work and optimization.
  • Experience mentoring and upskilling peers.
  • Agile delivery experience.

Responsibilities

  • Lead the design and build of scalable data pipelines and models in Azure Databricks on a new platform.
  • Own pipelines end to end, from source-system ingestion through transformation and modeling to outputs for downstream analytics, BI, and AI/ML use cases.
  • Refactor and migrate multiple legacy data warehouses into the centralized Databricks lakehouse.
  • Develop daily in Python, SQL, and Spark, using Databricks Asset Bundles, Spark workflows, and API-driven jobs.
  • Establish engineering standards for CI/CD, automated data testing, monitoring, and alerting.
  • Help shape and enforce data modeling and governance standards.
  • Troubleshoot and optimize platform performance.
  • Mentor and coach junior engineers, conduct code reviews, and lead by example in design, coding, and testing practices.
  • Partner with Data Engineering, BI, AI, Finance, and product owners, translating technical work into business impact.

Benefits

  • Health insurance
  • 401(k)
  • Paid time off
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service