Senior Software Engineer

LanceDB • San Francisco, CA

1d • $180,000 - $250,000

About The Position

LanceDB is an AI-native Multimodal Lakehouse that enables researchers to curate large collections of video, audio, and other derived signals, and supports AI training and development. The Senior Software Engineer will help expand Lance and LanceDB within the broader data infrastructure ecosystem, working at the intersection of high-performance computing, big data, and open-source systems. The role focuses on scale and performance improvements, integrations with data and AI systems, simpler distributed operations, and improvements to usability and maintainability.

Requirements

• 10+ years of experience building high-performance databases, big data systems, or large-scale data services.
• Deep understanding of internals of open-source Big Data or AI training systems, such as Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX.
• Strong experience with high-performance computing in C++, Java, and/or Scala.
• Experience with Rust or willingness to learn it.
• Proven ability to move fast, work independently, and collaborate with a high-caliber team.

Nice To Haves

• Contributor, committer, or PMC member in Apache or other large open-source projects.
• Experience with Apache Arrow, DataFusion, Parquet, Iceberg, or Delta Lake.
• Track record of driving large features or integrations in distributed systems.
• Strong community presence and passion for open-source collaboration.

Responsibilities

• Design and maintain efficient distributed Lance dataset operations.
• Build efficient indices that enable predicate pushdown and accelerate queries in Spark, Ray, or Trino.
• Work on table formats, data encodings, and other aspects of the Lance format in Rust.
• Drive open-source community efforts to integrate the Lance format with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems.
• Operate and improve internal data processing infrastructure.
• Promote the Lance format in open-source communities and at Big Data conferences.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed