The familiar browser download has a simple shape: one response becomes one file. That shape works until the thing being downloaded is not really a file at all. At Aina, a dataset can be a directory tree containing terabytes of footage. Asking the browser to download it exposes a mismatch between the interface and the thing a person actually wants.
There are a few conventional ways to hide that mismatch. The browser can start a separate download for every object, but thousands of independent downloads are difficult to supervise and do not reliably reconstruct a directory tree. The application can collect the data in memory, but the browser is the wrong place to assemble a multi-terabyte blob. The server can put everything into a zip file, which makes a folder look like the one file the download interface expects.
Zip is appealing because it solves the shape of the problem. It is less appealing when we look at what the bytes have to do.
A server can build the archive on demand and stream it without holding the whole result in memory. Now, however, every download is a new composite response. The individual source objects may already be cached at the edge, but the archive itself has to be assembled as one continuous stream. If the transfer stops near the end, resuming that stream is also more complicated than requesting an ordinary object again.
Prebuilding archives changes the tradeoff rather than removing it. For a long time, Aina kept pre-zipped copies of datasets. Those copies were easy to download and easy to cache, but they duplicated storage, needed their own lifecycle, and became stale whenever the underlying folder changed.
There was not even a compression win to offset those costs. Our large objects are usually camera media, which is already compressed. Wrapping it in…
