Senior Data Engineer

Independence Pet Group • Chicago, MI

3d • Onsite

About The Position

Independence Pet Holdings, established in 2021, is a corporate holding company managing a diverse portfolio of pet health brands and services, including insurance, pet education, and lost recovery services across North America. We aim to simplify the pet parenting journey and improve pet well-being through innovative solutions. Clarus, a Doubtless company, is a pioneering pet health technology company focused on transforming pet healthcare. We are seeking a seasoned Senior Data Engineer to join our data team, responsible for building data pipelines across the medallion architecture. This role is crucial for supporting product development processes by designing, developing, and optimizing data pipelines, implementing data checks, and ensuring data accuracy and integrity.

Requirements

• Bachelor’s degree in computer science, Information Technology, or related field.
• At least 7 years of experience in building data pipelines using ELT/ETL.
• Experience working with Databricks, Spark, and Azure data services.
• Strong coding skills with Python, SQL and PySpark.
• Experience designing and maintaining config-driven pipelines with support for Change Data Capture (CDC) and event/stream-based processing.
• Proficiency with NoSQL and SQL databases.
• Strong understanding of data modeling, schema design, and database performance optimization.
• Practical experience working with various file formats, including JSON, Parquet, and ORC.
• Hands-on experience building and maintaining CI/CD pipelines (Azure DevOps, GitLab) and automating data workflow deployments.
• Solid understanding of data governance, lineage, and cloud security (Unity Catalog, encryption, access control).
• Excellent problem-solving and analytical skills.
• Experience with data modeling tools and techniques.
• Knowledge of cloud-based data solutions and platforms.

Responsibilities

• Design, develop, and deploy robust and scalable data pipelines using Databricks, integrating data from various sources.
• Optimize pipeline performance for greater efficiency and cost-effectiveness.
• Contribute to the design and implementation of robust data models in the data lake environment, ensuring data consistency, integrity and accessibility for various use cases.
• Implement rigorous data quality controls and validation procedures throughout the data pipeline to ensure high accuracy and reliability.
• Comply with the data governance policies and best practices.
• Seamlessly integrate Databricks with Azure cloud services for storage, compute and security, leveraging services like Azure Data Lake Storage.

Benefits

• Comprehensive full medical, dental and vision Insurance
• Basic Life Insurance at no cost to the employee
• Company paid short-term and long-term disability
• 12 weeks of 100% paid Parental Leave
• Health Savings Account (HSA)
• Flexible Spending Accounts (FSA)
• Retirement savings plan
• Personal Paid Time Off
• Paid holidays and company-wide Wellness Day off
• Paid time off to volunteer at nonprofit organizations
• Pet friendly office environment
• Commuter Benefits
• Group Pet Insurance
• On the job training and skills development
• Employee Assistance Program (EAP)

What This Job Offers

• Job Type: Full-time
• Career Level: Senior