Data Engineer II

Hagerty

3d•Remote

About The Position

Hagerty is seeking a Data Engineer II to build and maintain data pipelines and services for its internally developed Enterprise Data Hub (EDH), which includes data processing and storage, services, and APIs. The role develops data pipelines, services, and cloud-based infrastructure to support Hagerty’s insurance business and automotive lifestyle brand. The position works with engineers in an agile environment, using modern cloud technologies to support data-driven decision-making, analytics, and data products. It also involves collaboration with internal and external stakeholders, Data Quality Engineers, and Data Scientists to build, validate, and operate data solutions. The role is open to U.S. remote work. Team members living within 20 miles of Hagerty’s Traverse City headquarters follow a hybrid schedule, working in the office three days per week. Quarterly-event travel may be required. Compensation, benefits, and perks are available by contacting recruiting for applicants in the specified jurisdictions.

Requirements

Strong problem-solving abilities and attention to detail.
Effective written and verbal communication with varied stakeholders.
Experience delivering data products as productionizable software solutions.
Commitment to quality through rigorous code development, testing, automation, and other software engineering best practices.
Experience developing solutions using Python and cloud-based infrastructure such as AWS, Azure, or GCP.
Experience with imperative ETL design, implementation, and maintenance (e.g., Apache Airflow or NiFi), or declarative ETL tools (e.g., Informatica, Talend, or Pentaho).
Functional knowledge of relational databases and SQL query authoring.

Nice To Haves

Associate degree, preferably in a technical or analytical field, or relevant work experience.
Additional 3+ years working in another role within an IT delivery team, such as developer, engineer, data analyst, quality assurance analyst, ETL developer, or DBA.
Experience developing infrastructure as code in a cloud environment; Terraform experience is preferred.
Experience cataloging and processing non-relational data.
Experience with open-source data processing technologies, such as Kafka or SQS streaming services, MapReduce or Spark processing frameworks, and EMRFS or HDFS file stores.
Experience evaluating data containers for workload needs, such as JSON, delimited files, Avro, and Parquet.
Experience with container-based development.
Familiarity with public-company requirements, including Sarbanes-Oxley and applicable regulations.

Responsibilities

Apply software development and big data engineering best practices.
Develop and implement robust, scalable data pipelines using Python, SQL, parallel processing frameworks, and AWS or Salesforce cloud solutions.
Develop and implement batch data pipelines using tools such as Apache Airflow, Snowflake, and AWS products including EC2, Fargate, ECS, Lambda, and RDS.
Develop streaming data integrations for products across Hagerty and support real-time reporting.
Develop Enterprise Data Hub platform infrastructure using Terraform infrastructure-as-code.
Develop and support Hagerty’s cloud-based data warehouse for analytics and product reporting.
Partner with internal and external stakeholders to gather requirements, recommend solutions, and productionize data ingestions and analytic workloads.
Develop solutions to catalog and manage metadata for data governance and democratization.
Partner with Data Quality Engineers to define and implement automated tests and data reconciliation for ETL processes and data quality and integrity.
Mentor junior team members in software and big data engineering practices.
Partner with Data Scientists to design, code, train, test, deploy, and iterate machine-learning algorithms and systems at scale.
Create and share technical artifacts and documentation to support development and maintenance of data products.
For SOX-compliant roles, design, execute, and document assigned internal controls and attest to applicable financial reporting data completeness, accuracy, and compliance.