Comparison
s3fs vs Training Pipes
s3fs is a fine prototype tool. In a real ML training pipeline, it falls over. Here's what you lose with FUSE and what you get from a proper NFS gateway instead.
Short answer
s3fs-fuse implements a filesystem-like view of an S3 bucket using FUSE — every filesystem operation on every node becomes an HTTP request to S3, with limited POSIX semantics and no shared cache. For exploration and light scripts, it works. For GPU training that re-reads datasets and checkpoints large models, POSIX breaks silently, costs compound per-request, and tail latencies crush GPU utilization. Training Pipes runs a real NFS gateway with an NVMe cache between your compute and object storage, so your training code sees a proper filesystem backed by a shared cache.
Feature by feature
s3fs and Training Pipes, side by side
| Feature | s3fs | Training Pipes |
|---|---|---|
| Protocol | FUSE → HTTP (S3 API) | NFSv4 → gateway → S3 |
| POSIX atomic rename | Emulated (unsafe) | Yes |
| POSIX file locking | No-op | Yes |
| mmap support | Unreliable | Yes |
| Shared cache across nodes Shared cache is the biggest difference for multi-node training. | No | Yes |
| Prefetch / preload | No | Yes |
| Container/Kubernetes support | Requires privileged pods | Standard NFS CSI |
| Per-read S3 request cost | Every read on cold cache | Only on cache miss |
| Tail latency under load | 100ms-1s+ | Sub-ms on cache hit |
| Works with BYO S3/GCS/Azure | S3-compatible only | Yes |
| Managed service | No | Yes |
When to use s3fs
- Ad-hoc scripts on a dev workstation.
- Exploring bucket contents interactively.
- One-off data loads where latency doesn't matter.
- Single-node jobs with short runtime.
When to use Training Pipes
- Multi-node GPU training.
- Workloads that re-read datasets across epochs.
- Anywhere POSIX semantics (locks, atomic rename, mmap) actually matter.
- Kubernetes environments where privileged FUSE pods are off-limits.
- Any time S3 request and egress charges are showing up on your bill.
The verdict
If you're running training workloads on cloud GPUs and your data lives in object storage, s3fs is the wrong tool. A real NFS gateway with caching gives you higher GPU utilization, correct POSIX semantics, and dramatically lower egress/request costs. Keep s3fs for dev boxes and ad-hoc scripts — not for the hot path of training.
FAQ