Data Platform Engineer, Ingestion

Minerva•New York City, NY

21h•Onsite

About The Position

Minerva is hiring a mid-level individual contributor for its data platform team, reporting to the Head of Data Platform. The role owns how customer and vendor data arrives and remains dependable, building ingestion pipelines for sources such as APIs, files, database extracts, and approved scraping sources. The engineer will automate source onboarding and create reusable modules that other Minerva teams can use for managed ingestion, storage, and monitoring without operating platform infrastructure themselves. The work involves handling high-volume, inconsistent data and changing schemas; building safe incremental loads, backfills, and retries; quarantining bad records; controlling throughput and processing cost; carrying source metadata through pipelines; and enforcing privacy controls. The engineer will own integrations in production, connect them to shared monitoring, maintain recovery runbooks, and resolve source-specific failures. The stack includes Dagster and Airflow for orchestration; dbt and SQLMesh for transformation; Snowflake and a lakehouse built on Spark, Iceberg, Trino, and AWS Glue for storage; and Python on AWS. The position is based in person at Minerva's office at 300 Kent Avenue in Williamsburg, Brooklyn.

Requirements

Typically three or more years building and supporting production data pipelines, or equivalent ownership.
Experience bringing in data from many vendors and source types, including messy data at scale, and supporting it after launch.
Practical judgment about incremental processing, backfills, failures, and cost, with the ability to demonstrate data correctness as volume grows.
Experience automating repeated work or building components that other engineers used.
Strong Python and SQL skills.
Ability to clarify source owners' needs and document what the system will and will not support.
Located in New York City or willing to relocate.
Authorization to work in the United States; visa sponsorship is considered case by case.

Nice To Haves

Experience with connector frameworks, whether off the shelf or internally built.
Experience with onboarding automation or self-service data tooling.
Experience with consumer, identity, or contact data and the associated privacy requirements.
Production experience with Dagster, Iceberg, or Snowflake.

Responsibilities

Build high-volume ingestion pipelines for APIs, files, database extracts, and approved scraping sources with inconsistent formats and changing schemas.
Make incremental loads, backfills, and retries safe so they do not duplicate or corrupt data; quarantine bad records and control throughput and processing cost.
Automate source onboarding from configuration and access checks through provisioning, validation, deployment, and first delivery.
Build reusable connectors and storage components with tests, examples, and clear interfaces for other Minerva teams.
Carry source metadata through pipelines so the origin and delivery history of each value can be identified.
Enforce privacy controls that prevent restricted data from reappearing through new loads or replays.
Own integrations in production by connecting them to shared monitoring, writing recovery runbooks, and fixing source-specific problems.
Measure source onboarding effort and test recovery; improve time to onboard sources and reduce hands-on feed recovery.