About The Position

The Production Engineer will work with the customer's Cloud Foundation AI Services team, which builds and operates storage infrastructure that enables AI researchers to train models reliably and efficiently across cloud environments. The team connects high-performance filesystems, object storage, and data-management services to AI compute so training datasets and model checkpoints are available where needed. This hands-on role focuses on turning demanding AI workloads into reliable, scalable infrastructure. The work spans software development, performance analysis, automation, and production ownership. The position is 100% remote and open to candidates in the United States. The posting lists 2–5 openings.

Requirements

  • More than 10 years of experience.
  • Experience building or operating production distributed storage, high-performance filesystems, or large-scale data infrastructure.
  • Strong Linux skills and ability to debug filesystem, network, and concurrency issues in production.
  • Ability to write maintainable software or automation in a language such as Python, Go, C++, or Rust.
  • Experience owning a reliability or performance improvement end to end, from identifying the problem through deployment and verification.

Nice To Haves

  • Experience with cloud infrastructure and Kubernetes, or a comparable large-scale orchestration environment.
  • Depth in NFS/POSIX and distributed-filesystem internals, including metadata, caching, consistency, availability, and recovery.
  • Hands-on experience with one or more distributed or network storage systems such as Hammerspace, Lustre, ZFS/OpenZFS, VAST, or Pure Storage; comparable systems also count.
  • Experience with AI or HPC storage patterns, including high-throughput data loading, checkpoint writes and restores, metadata-heavy access, and staging data for accelerator clusters.
  • Experience with infrastructure-as-code and observability tooling, such as Terraform, Helm, Kubernetes operators, metrics, tracing, and benchmarking.
  • Experience with object/S3 storage, tiering, and cross-region movement.

Responsibilities

  • Build and operate distributed file and object storage for AI training, including shared researcher filesystems, training datasets, and model checkpoints.
  • Diagnose and improve end-to-end storage performance across metadata, throughput, latency, caching, and network behavior.
  • Automate the storage lifecycle, including provisioning, scaling, upgrades, and recovery, using Kubernetes and infrastructure-as-code.
  • Move and migrate data across clusters, regions, and clouds while maintaining data integrity and minimizing disruption to researchers.
  • Maintain reliability through observability, alerting, testing, safe rollouts, and incident response.
  • Partner with research, security, and storage-provider teams.

Benefits

  • Medical benefits options
  • Dental benefits options
  • Disability insurance
  • Paid time off, inclusive of sick leave
  • Other paid and unpaid leave options
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service