Data Platform Engineer, Ingestion

Minerva•New York, NY

22h•$150,000 - $236,000•Onsite

About The Position

Minerva is hiring a mid-level individual contributor for its data platform team to build dependable ingestion pipelines for customer and vendor data. The engineer will bring in data from APIs, files, database extracts, and approved scraping sources, including high-volume sources with inconsistent formats and changing schemas. The role focuses on making ingestion safe to rerun, automating source onboarding, and building reusable connectors and storage components that other teams can use without managing platform infrastructure themselves. The engineer will own integrations in production, including monitoring, recovery runbooks, and source-specific fixes. The role also requires preserving source metadata throughout pipelines and enforcing privacy controls so restricted data cannot re-enter through a new load or replay. The data product team determines which sources matter and what constitutes good data; this role owns how data arrives and remains dependable. The position reports to the Head of Data Platform and is based in person at the company's Williamsburg, Brooklyn office. Candidates must be in New York City or willing to relocate and must be authorized to work in the United States. Visa sponsorship is considered case by case. The technology stack includes Dagster and Airflow for orchestration; dbt and SQLMesh for transformation; Snowflake and a lakehouse using Spark, Iceberg, Trino, and AWS Glue for storage; and Python on AWS for code. Compensation is a base salary of $150,000–$236,000, depending on level, plus meaningful equity. The company describes the salary range as its good-faith range for the New York City role at the time of posting; offer placement depends on level and interview experience.

Requirements

Typically three or more years building and supporting production data pipelines, or equivalent ownership.
Experience bringing in data from many vendors and source types, including messy data at scale, and supporting it after launch.
Practical judgment about incremental processing, backfills, failures, and cost, with evidence that data remained correct as volume grew.
Experience automating repeated work or building components used by other engineers.
Strong Python and SQL skills.
Ability to determine what a source owner needs and document what the team will and will not support.
Based in New York City or willing to relocate.
Authorization to work in the United States; visa sponsorship is considered case by case.

Nice To Haves

Experience with connector frameworks, either off-the-shelf or custom-built.
Experience with onboarding automation or self-service data tooling.
Experience with consumer, identity, or contact data and related privacy requirements.
Production experience with Dagster, Iceberg, or Snowflake.

Responsibilities

Build high-volume ingestion pipelines for APIs, files, database extracts, and approved scraping sources, handling inconsistent formats and changing schemas.
Design incremental loads, backfills, and retries so reruns do not duplicate or corrupt data; quarantine bad records and manage throughput and processing costs.
Automate source onboarding from configuration and access checks through provisioning, validation, deployment, and initial delivery.
Build reusable connectors and storage components with tests, examples, and clear interfaces for use by other Minerva teams.
Preserve source metadata, including vendor, delivery, and timing information, throughout pipelines.
Enforce privacy controls that prevent restricted data from re-entering through new loads or replays.
Connect integrations to shared monitoring, create recovery runbooks, and resolve source-specific production issues.
Measure source onboarding time and the frequency of feeds requiring hands-on recovery.