Comparison
FSx for Lustre vs Training Pipes
FSx for Lustre is a high-performance parallel filesystem for HPC. Training Pipes is a caching gateway over object storage for ML. Both give you a file system; the economics and operational model are very different.
Short answer
FSx for Lustre is AWS's managed Lustre filesystem, designed for HPC workloads that need extreme parallel throughput on a single job. It links to S3 for data import/export but isn't a cache over S3 — it's a separate filesystem that you provision, pay for by capacity, and tear down when done. Training Pipes keeps your data in object storage permanently and uses a regional NVMe cache to accelerate reads; you don't provision and tear down filesystems per job.
Feature by feature
FSx Lustre and Training Pipes, side by side
| Feature | FSx Lustre | Training Pipes |
|---|---|---|
| Filesystem type | Lustre (parallel FS) | NFSv4 via gateway |
| Peak single-job throughput FSx wins on raw throughput for a single HPC job; TP wins on cost for training. | 12+ GB/s per TB provisioned | 1-4 GB/s per file system (NVMe-limited) |
| Provisioning model | Provision filesystem per job | Persistent, always-on |
| Pricing model | Per-TB-hour | Plan tier + object storage |
| Data durability | Backed by S3 link | Object storage is the primary |
| Multi-cloud | No | Yes |
| Bring your own bucket | S3 data repository | AWS, GCS, Azure, R2, Wasabi, MinIO |
| POSIX semantics | Yes | Yes |
| Shared cache across jobs | No | Yes |
When to use FSx Lustre
- Classic HPC workloads (CFD, seismic, genomics) with true parallel-filesystem needs.
- Single jobs that need sustained multi-GB/s throughput.
- Batch workloads where provision-and-tear-down is the operational model.
- You're already deep in the AWS HPC ecosystem (ParallelCluster, Batch).
When to use Training Pipes
- Training ML models with persistent shared datasets.
- Multiple concurrent training jobs against the same data.
- Multi-cloud or BYO-storage requirements.
- Workloads where a persistent caching layer produces more ROI than per-job filesystem provisioning.
- You want both NFS and the S3 API to the same dataset.
The verdict
Pick FSx for Lustre if you're running classic HPC workloads (CFD, genomics, seismic) where Lustre semantics and multi-GB/s single-job throughput justify the per-hour cost. Pick Training Pipes if you're doing ML training with shared datasets, care about persistent storage economics, need multi-cloud or BYO, or run many jobs against the same data where a shared cache compounds.
FAQ