Object storage, NFS, and POSIX access
How object storage behaves under training workloads, where POSIX semantics matter, and what changes when the same bucket is reachable over both NFS and the S3 API.
11 posts
You already have data in S3, GCS, R2, or Wasabi. Here's how to bring existing cloud storage into a unified AI-ready storage layer without migration, and why you'd want to.
NFS dominates in Linux-first ML shops; SMB dominates in mixed Windows environments. Here's how to choose, and why enterprise AI teams often end up wanting both.
When five engineers each copy the same 20TB dataset into ephemeral storage, you've got a problem. Here's how to share datasets efficiently across teams and runs.
You don't notice egress until you see the bill. Here's how ML training pipelines quietly rack up cross-region transfer costs, and the architecture that fixes it.
Synthetic storage benchmarks lie about what DataLoader performance feels like in practice. Here's how to measure what your training pipeline actually cares about.
POSIX semantics on top of object storage is an old and messy problem. Here's what's possible, what's impossible, and what ML teams should actually demand from a storage layer.
Searching for 'mount S3 as NFS' turns up a dozen FUSE-based tools. Here's why none of them survive production ML workloads, and what actually works.
EFS gives you POSIX on AWS. Training Pipes gives you POSIX plus caching plus any cloud. We ran the numbers on a realistic ML training workload to see which wins.
s3fs-fuse is a fine prototype tool and a dangerous production dependency. Here's what breaks, why, and what to use instead for real ML training workloads.
NFS and S3 solve different problems — but AI teams have to use both. Here's a clear framework for when each protocol wins, and how to stop choosing between them.
Object storage like S3 is cheap, durable, and infinite — but it isn't a filesystem. Here's why ML teams keep hitting the same wall, and what to do about it.