Staff Software Engineer, Data Infrastructure

Cohere•New York, NY
•$275,000 - $335,000•Remote

About The Position

Cohere’s Data Infrastructure team builds the storage and data movement layer that supports model training and evaluation. The team is developing a unified storage system to serve petabytes of training data and model checkpoints across multiple training clusters, with the goal of keeping thousands of GPUs busy. The Staff Software Engineer will architect and lead the design of this system from the ground up, driving its technical vision from concept through production. The role includes deploying and operating the platform across Kubernetes clusters at petabyte scale, defining organizational standards and best practices, and working with researchers and training-infrastructure teams to translate workload patterns into platform requirements. The engineer will address networking, I/O, and consistency challenges involved in moving large datasets and checkpoints across regions and backends, with GPU idle time and time-to-insight as key success measures.

Requirements

  • Strong storage fundamentals, including replication, consistency, caching, and data lifecycle management, with demonstrated architecture and design experience.
  • Strong coding ability in Python and Go, with the ability to lead implementations and mentor others in either language, and willingness to contribute to the other.
  • Proven experience running stateful systems on Kubernetes, including Persistent Volumes, CSI drivers, and StatefulSets, and the capacity to design and execute operational strategies.
  • Hands-on experience with cloud object storage such as S3 and POSIX-style filesystems.

Nice To Haves

  • Experience with parallel or HPC filesystems such as Weka, VAST, or Lustre, and a track record of applying them at scale.
  • Familiarity with data-loading and checkpointing patterns used in large-scale model training, with the ability to innovate on them.
  • Interest in how large models are trained and evaluated, including the trajectory and evaluation data those runs generate, and a desire and capacity to guide others in understanding the workloads the platform serves.

Responsibilities

  • Design, architect, and operate the distributed storage system that feeds model training and evaluation, driving its technical vision from concept through production.
  • Lead and own deployment of the system across multiple Kubernetes clusters at petabyte scale, setting organizational standards and best practices.
  • Collaborate with researchers and training-infrastructure teams to define how jobs read and write data, and translate those patterns into throughput, latency, and durability requirements for the platform roadmap.
  • Address networking, I/O, and consistency challenges in moving large datasets and checkpoints across regions and backends.
  • Drive solutions that reduce GPU idle time and improve time-to-insight.

Benefits

  • Weekly lunch stipend of $75/£75 or equivalent in local currency.
  • Full health and dental benefits, including a separate mental health budget.
  • RRSP matching, 401(k), and Pension Scheme.
  • 100% parental leave top-up for up to six months, for either parent.
  • Annual enrichment benefits for arts and culture, fitness and wellness, quality time, and workspace improvements.
  • Education and learning stipend for conferences, courses, and coaching.
  • Six weeks of paid vacation (30 working days).
  • Travel budget for remote employees to visit other offices, plus an annual company offsite.
  • Daily lunch program, snacks, and regular community and social events for employees in an office.
  • Co-working benefit for employees who are not near an office.
  • $500 home office stipend.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service