Senior Data Engineer

Foresight Works Ltd•San Francisco, CA

8h•$180,000 - $240,000

About The Position

Foresight is hiring a Senior Data Engineer to build the data foundation behind its platform for complex capital projects. This hands-on role on a new team focuses on turning complex project data from customers’ existing tools into clean, trusted, well-modeled data that supports predictions, AI recommendations, analytics, and customer reporting. The role treats data engineering as product engineering: the data engineer is responsible for making the underlying data correct, explainable, and available when needed. The work may include ingesting project data from scheduling and planning tools, preserving a reliable history of updates, preparing features and datasets for machine learning and AI, building analytics and reporting pipelines, and implementing data-quality checks, lineage, access controls, and regional data handling. The engineer will own pipelines from design through production, collaborate with ML and full-stack engineers, design for safe retries and recovery, monitor data health, manage costs, and help establish the team’s data standards and platform.

Requirements

At least 5 years of experience building and operating production data pipelines, with ownership from source to consumer.
Strong Python skills and advanced SQL skills, including writing and tuning complex queries.
Deep data-modeling experience in Postgres or a similar relational database, including schema design, indexing, partitioning, and migrations.
Experience parsing and normalizing messy real-world data, including large files, inconsistent formats, and missing or contradictory values.
Hands-on experience with workflow orchestration for multi-step pipelines, including dependencies, retries, and backfills.
Experience with AWS data services such as S3, managed databases, streaming, and serverless compute.
A testing mindset for data, including validation, quality checks, and reproducible, idempotent pipelines.
Clear communication skills and comfort making pragmatic trade-offs on a small team.

Nice To Haves

Experience preparing data for machine learning, including feature pipelines, training datasets, and maintaining consistency between offline and online data.
Experience with embeddings and vector search, such as pgvector or a search engine.
Experience with event-driven data, including change data capture, streaming platforms, and the outbox pattern.
Familiarity with project scheduling data or tools used in construction, engineering, or energy.
Experience working with regulated enterprise or public-sector data, including SOC 2, GDPR, or data residency.
Experience with infrastructure as code and CI/CD for data pipelines.
Early-stage or founding-team experience.

Responsibilities

Own data pipelines from design through production, define data contracts, set freshness and quality targets, and ensure those targets are met.
Build robust ingestion pipelines for project data from scheduling and planning tools, including handling messy and inconsistent files.
Model data to preserve each update as received and enable comparisons between points in time.
Build features, training datasets, and search indexes for predictions and AI recommendations, keeping them fresh as new data arrives.
Build data models and pipelines for customer dashboards, stakeholder reports, exports, and internal analytics.
Implement automated data-quality checks, lineage, and access controls to support accuracy, customer data isolation, and regional data requirements.
Partner with ML engineers on model features and datasets, and with full-stack engineers on data supporting product features.
Design pipelines to retry safely, support reruns without duplicating data, and recover from bad inputs.
Make data observable through monitoring that catches problems before customers notice them.
Manage costs as data volumes grow and choose appropriately simple tools.
Help establish data standards and shape the platform as one of the first data engineers on the team.