AI Storage Solutions Expert

Bitdeer Technologies Group • San Jose, CA

10d • Onsite

About The Position

Bitdeer is building an AI-operated GPU cloud and is seeking an AI Storage Solutions Expert to own the IO layer that trains AI models. This role is crucial for ensuring the performance and reliability of storage systems, which are critical for AI workloads. The expert will deploy and operate high-performance storage for AI training and inference across US data centers, implement multi-tenant storage solutions with QoS and access controls, and optimize GPU Direct Storage. Additionally, the role involves feeding the AIOps substrate with telemetry data to predict and prevent storage faults before they impact critical training runs. Success in the first year includes establishing observability, a baseline predictor for top storage fault classes, reducing MTTR for storage incidents, and contributing to the storage design for new GPU clusters.

Requirements

• 5+ years in enterprise or HPC storage operations, with at least 2 years supporting AI/ML workloads
• Hands-on deployment and operations experience with at least two of: WEKA, VAST Data, Ceph, DDN/Lustre
• Strong understanding of AI training I/O patterns: checkpoint frequency, dataset loading, shuffle buffers
• Experience with high-performance storage networking (NFS over RDMA, NVMe-oF)
• Knowledge of GPU Direct Storage and RDMA-based data transfer
• Proficiency in storage performance benchmarking and tuning (fio, IOR, mdtest)
• Experience implementing multi-tenant storage with isolation and QoS
• Strong Linux systems knowledge (kernel tuning, filesystem internals, block device management)
• Instinct for turning ops toil into ML signal — you've either shipped an anomaly detector for storage/IO telemetry or you can articulate the labels and features you'd need to.
• Runbook-as-code mindset — every SOP you write should be executable by a machine within a quarter.

Responsibilities

• Deploy and operate parallel/distributed storage systems: WEKA, VAST Data, Ceph, DDN/Lustre.
• Design storage architectures optimized for AI workload patterns — checkpoint I/O bursts, sequential dataset reads, KV cache for inference.
• Implement multi-tenant storage isolation with per-tenant QoS, quotas, and access controls; configure and optimize GPU Direct Storage for direct GPU-to-storage data paths.
• Deploy and manage storage networking (NFS over RDMA, NVMe-oF, high-speed storage fabrics) and Nvidia CMX for cluster-wide storage orchestration.
• Diagnose and tune storage performance: IOPS, throughput, latency profiling with fio, IOR, mdtest; own the runbook for common failure modes.
• Plan storage capacity aligned with GPU cluster growth and customer workload projections; manage firmware, data migration, and DR procedures.
• Instrument storage telemetry — IO tail latency, checkpoint durations, NVMe SMART, filesystem health, RDMA counters — into the metrics/logs/traces store the platform team runs.
• Partner with the platform team to define the storage-fault predictor: which signals, which labels (from your incidents), which false-positive tolerances.
• Convert every novel incident into an automation: SOPs become runbook-as-code, runbook-as-code becomes an agent-executable remediation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed