Senior Data Engineer
About The Position
Retail Insight is seeking an experienced Senior Data Engineer to build and support reliable, scalable data systems powering solutions for US retail clients. As the US client portfolio grows, the role will work with complex data environments and translate business requirements into practical technical solutions. The engineer will work closely with the Chicago-based Lead Data Engineer, data scientists, analysts, product teams, colleagues, and client stakeholders to deliver data pipelines and products that help clients make better decisions. This is a hands-on role combining technical expertise with collaboration and clear communication. The successful candidate will contribute to the evolution of the data platform, improve reliability and performance, maintain high engineering standards, and support successful delivery.
Requirements
- Solid, demonstrable experience in data engineering, including building and maintaining production-grade Databricks pipelines.
- Experience with Databricks Unity Catalog and Databricks Asset Bundles.
- Strong hands-on knowledge of Apache Spark, Delta Lake, and Databricks Workflows.
- Proficiency in Python and SQL, with a good understanding of data modelling, ETL/ELT patterns, and distributed data processing.
- Experience with Azure data platforms, including Azure Data Factory, ADLS Gen2, and Azure DevOps, or equivalent services on AWS or GCP.
- Experience implementing data quality checks and pipeline monitoring.
- Strong communication skills and the ability to work effectively with technical and non-technical colleagues.
- A collaborative approach, ownership of quality outcomes, and a drive to learn and improve.
- A degree in computer science, data engineering, or a related field, or equivalent practical experience.
- Willingness to travel as required.
Nice To Haves
- Familiarity with infrastructure-as-code tools such as Terraform or Bicep.
- Familiarity with dbt or similar data transformation frameworks.
- Exposure to real-time streaming technologies such as Kafka or Event Hubs.
- Exposure to retail or consumer packaged goods (CPG) data, such as point of sale, inventory, or supply chain.
- Experience working directly with clients.
Responsibilities
- Build and maintain scalable data pipelines, including developing and optimizing batch and streaming workflows for reliability and performance.
- Contribute to the design and implementation of data architecture.
- Implement data quality checks, monitoring, and observability across pipelines.
- Apply security, access control, and governance standards established by the platform team.
- Collaborate with data scientists and analysts to deliver data products that support their work.
- Work with the Lead Data Engineer to translate business requirements into technical solutions.
- Participate in code reviews, uphold engineering quality, and contribute to team standards.
- Identify opportunities to optimize cloud costs and platform performance.
- Contribute to platform documentation, tooling choices, and continuous improvement.
- Work with colleagues and client stakeholders to understand their needs and support successful delivery.
- Travel as required.
Benefits
- Hybrid and flexible working, including work-from-anywhere benefits, summer hours, and extra days off for life’s big moments, including birthdays and weddings.
- 26 days of PTO, accrued with service.
- Paid volunteer days.
- Mental health resources and wellbeing support.
- Access to a learning platform, career progression plans, and support to create a home workspace.
- Comprehensive medical, dental, and vision coverage.
- 6% matched retirement plan.
- Annual phone reimbursement.
- Onsite gym access.
- Customizable benefits via an online platform.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
