Shritesh Bhattarai

The difficult part is writing the distributed cache that will sit in front of object storage to lower the latency and reduce cost. Getting it to work is trivial. Using object store for WAL or data is trivial to get going. You have to pay the piper. The piper collects on the write/durability path, and the currencies are coupled. Object storage gives you durability at PUT latency (tens to low-hundreds of ms with an ugly tail on standard S3) and bills per request, PUTs roughly 10× GETs. So WAL-on-S3 silently pins everything to the worst-case: commit latency is PUT latency, and every small write is a billed PUT. The cost that bites isn’t $/GB, it’s request count and cross-AZ transfer. Tiny objects blow up the request bill, so you’re forced into compaction/batching to build big objects, but batching has its own read/write amplification against S3, so you’ve relocated the request bill, not killed it. Zone aligned reads end up mattering as much as hit rate. It’s the classic latency/bandwidth tradeoff with cost dimensions.