Storage for training pipelines
DataLoader throughput, checkpoint write patterns, dataset sharing across runs, and the egress costs that show up once your pipeline spans regions.
8 posts
EBS, EFS, FSx, object storage, CSI drivers — Kubernetes gives you many options for ML storage and all the wrong defaults. Here's the pattern that actually works for training workloads.
When five engineers each copy the same 20TB dataset into ephemeral storage, you've got a problem. Here's how to share datasets efficiently across teams and runs.
You don't notice egress until you see the bill. Here's how ML training pipelines quietly rack up cross-region transfer costs, and the architecture that fixes it.
Writing a 500GB checkpoint every hour stresses your storage in ways that training data doesn't. Here's how to design a checkpoint pipeline that's fast, reliable, and doesn't cost a fortune.
Synthetic storage benchmarks lie about what DataLoader performance feels like in practice. Here's how to measure what your training pipeline actually cares about.
A caching gateway colocated with your GPUs is the biggest single lever for training throughput. Here's how the architecture works and why it produces such dramatic speedups.
Step-by-step guide to mounting cloud storage as a filesystem on your GPU nodes, without the usual FUSE pain or EFS sticker shock.
Object storage like S3 is cheap, durable, and infinite — but it isn't a filesystem. Here's why ML teams keep hitting the same wall, and what to do about it.