Senior Data Engineer

Foresight Works Ltd•New York, NY

7h•$180,000 - $240,000

About The Position

Foresight is hiring a Senior Data Engineer to build the data foundation behind its product. The role involves creating pipelines that transform complex project data from customers’ existing tools into clean, trusted, well-modeled data for predictions, AI recommendations, analytics, and customer reporting. This is a hands-on role on a new team, focused on ensuring data is correct, explainable, and available when needed. The engineer will build ingestion pipelines for industry scheduling and planning tools, preserve update history so teams can compare data across time, prepare features and datasets for machine learning and AI, support analytics and reporting, and develop quality checks, lineage, and access controls. Customer data must remain accurate, isolated, and in the appropriate region. The position owns pipelines from design through production, partnering with ML and full-stack engineers. The engineer will design resilient, observable, and cost-conscious systems, contribute data standards, and help shape the platform as one of its first data engineers. Preferred locations are New York, San Francisco, and Montreal; occasional travel may be required. Total compensation is $180,000–$240,000, based on experience, level, and location.

Requirements

At least 5 years of experience building and running production data pipelines, with ownership from source to consumer.
Strong Python skills and advanced SQL skills, including writing and tuning complex queries.
Deep data modeling experience in Postgres or a similar relational database, including schema design, indexing, partitioning, and migrations.
Experience parsing and normalizing messy real-world data, including large files, inconsistent formats, and missing or contradictory values.
Hands-on experience with workflow orchestration for multi-step pipelines involving dependencies, retries, and backfills.
Experience with AWS data services such as S3, managed databases, streaming, and serverless compute.
A testing mindset for data, including validation, quality checks, and reproducible, idempotent pipelines.
Clear communication skills and comfort making pragmatic trade-offs on a small team.

Nice To Haves

Experience preparing data for ML, including feature pipelines, training datasets, and maintaining consistency between offline and online data.
Experience with embeddings and vector search, such as pgvector or a search engine.
Experience with event-driven data, including change data capture, streaming platforms, and the outbox pattern.
Familiarity with project scheduling data or tools used in construction, engineering, or energy.
Experience with regulated enterprise or public-sector data, including SOC 2, GDPR, or data residency.
Experience with infrastructure as code and CI/CD for data pipelines.
Early-stage or founding-team experience.

Responsibilities

Own data pipelines from design through production, define data contracts, set freshness and quality targets, and ensure those targets are met.
Partner with ML engineers on model features and datasets, and with full-stack engineers on data supporting product features.
Design pipelines to retry safely, run again without duplicating data, and recover cleanly from bad inputs.
Make data observable and use monitors to catch problems before customers notice them.
Manage costs as data volumes grow by choosing the simplest effective tools.
Help establish the team's data standards and shape the data platform.