Software Engineer, Data Infrastructure

Thinking Machines Lab•San Francisco, CA

7d•$350,000 - $475,000•Onsite

About The Position

Thinking Machines is seeking an engineer to join their data infrastructure team. This role involves architecting and scaling the core infrastructure for distributed training pipelines, multimodal data catalogs, and intelligent processing systems that handle petabytes of data. The engineer will work closely with researchers to accelerate experiments, develop new datasets, improve infrastructure efficiency, and enable key insights. The ideal candidate is excited by distributed systems, large-scale data mining, open-source tools like Spark, Kafka, Beam, Ray, and Delta Lake, and enjoys building from the ground up. This is an evergreen role, meaning applications are continuously reviewed for current and future opportunities.

Requirements

Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
Proficiency in at least one backend language (Python or Rust).
Fluent in distributed compute frameworks such as Apache Spark or Ray.
Deeply familiar with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
Comfort operating across the stack and owning projects end-to-end.
Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

Nice To Haves

Hands-on experience with Kafka, dbt, Terraform, and Airflow.
Experience building a web crawler.
Extensive experience understanding and scaling deduplication, data mining, and search.
Strong knowledge of file formats and storage systems (e.g., Parquet, Delta Lake, etc.) and how they impact performance and scalability.
Proactive about documentation, testing, and empowering your teammates with good tooling.

Responsibilities

Design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities.
Develop high-throughput systems for data ingestion, processing, and transformation — including training data catalogs, deduplication, quality checks, and search.
Build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.
Implement and maintain monitoring and alerting to support platform reliability and performance.
Collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles.