Software Engineer, Data Infrastructure (Staff)

Lightfield•Cambridge, MA

2d•$180,000 - $300,000•Hybrid

About The Position

Lightfield is an AI-native CRM that automatically organizes customer interactions from email, calendar, and meetings into context like accounts, tasks, and insights. Backed by prominent investors and founded by individuals with prior success in building widely used products, Lightfield aims to revolutionize CRM by adapting to how companies work rather than forcing rigid systems. The company is experiencing rapid growth, leading to scaling pressures across its backend, infrastructure, and data systems. This role is crucial for building the next generation of data systems to support this growth. The current technology stack includes Postgres for the system of record, a sharded transactional outbox for change events, Redis for buffering, Typesense for search, BullMQ for processing, and Postgres for customer-facing analytics with row-level security. The next phase involves evolving this foundation to incorporate best practices in data architecture, including change data capture, event modeling, schema design, query performance optimization, freshness guarantees, and defining the boundaries between transactional and analytical workloads. A unique aspect of the system is its schema-flexible, graph-shaped data model (entity-attribute-value with typed edges) to accommodate customer-defined objects, attributes, and relationships at runtime, presenting complex challenges in schema design, indexing, and query performance. The role's scope extends beyond analytics to include customer-facing dashboards, historical and audit data, data for pipeline-generation products, and AI agent evaluation data. This is a data infrastructure role focused on building reliable foundations for product and engineering teams.

Requirements

Strong software engineering fundamentals.
Experience owning production data systems where query plans, replication lag, backfills, data freshness, schema evolution, or data correctness had real user-facing consequences.
Comfort debugging across multiple layers of the stack.
Good judgment about when to make a tactical fix and when to invest in a more durable platform or architecture change.
Product orientation: care about how data infrastructure decisions affect customers, users, and engineering velocity.
Clear communication, strong ownership, and a bias toward practical tradeoffs.

Nice To Haves

ClickHouse, OLAP systems, event pipelines, data warehouses, or analytical infrastructure.
Kafka, Flink, Spark, Iceberg, or similar streaming and lakehouse systems.
Postgres at scale, and the boundary between OLTP and OLAP systems.
APIs, queues, workflow systems, and distributed systems.
Observability, incident response, service ownership, and production debugging.
Data for ML/AI systems: enrichment pipelines, eval harnesses, or data-quality tooling.
Experience in a high-growth product environment.

Responsibilities

Scale the analytics engine behind customer-facing dashboards, addressing query performance under row-level security, workload isolation, read architecture, and observability.
Design ingestion paths, event models, schemas, and query patterns for moving data from transactional writes to search, dashboards, and history, ensuring freshness, correctness, replay, and failure recovery.
Evolve the schema-flexible, graph-shaped data model to maintain query performance for customer-defined objects, attributes, and relationships as they grow in size and complexity.
Build foundations for historical reporting and auditability, including attribute versioning, relationship history, and change capture.
Develop reliable data systems for usage metering, pipeline generation, and AI evaluation, where errors have direct customer, product, or financial impacts.
Determine when to leverage existing architecture and when to introduce new analytical, streaming, or workflow systems.
Set the technical direction, define abstractions, establish ownership boundaries, and implement engineering practices for data systems as the company scales.