How downloading works

Data is published one file per hour. Whatever you buy is a range over those files.

The manifest

Every order has a manifest — a JSON list of the files it covers, each with a path, size and sha256. Your client compares it against what is already on disk and fetches the difference. For a subscription the manifest grows as new hours are published, which is what makes "give me the freshest data" a single repeated command.

{
  "files": [
    {
      "id": 8412,
      "path": "raw/2026/08/24/13/data.parquet",
      "period_start": "2026-08-24T13:00:00Z",
      "size": 314572800,
      "sha256": "…",
      "url": "https://…/d/<token>/f/8412"
    }
  ]
}

The sync script

Bash and curl, nothing else. Downloads in parallel, resumes partial files, verifies checksums, skips what it already has.

curl -fsSL /client/fetch-dataset.sh -o fetch-dataset.sh
chmod +x fetch-dataset.sh
./fetch-dataset.sh <your manifest URL> ./data

# every hour, if you are on a subscription
./fetch-dataset.sh <your manifest URL> ./data

Options: -j N parallel downloads (default 4), --no-verify to skip hashing, --dry-run to see what it would fetch.

Resuming

Every file supports HTTP range requests. Published files are never modified in place — a corrected hour is published as a new revision under a new URL — so a partial download stays valid indefinitely and its ETag never changes under you. curl -C -, wget -c and aria2c all work. The authoritative checksum is the sha256 in the manifest; the sync script verifies every file against it.

The browser button

Fine for samples, aggregates and a single day. A 30-day raw window is hundreds of gigabytes across hundreds of files — use the script for that.

Why not rsync

rsync would need its own port and its own account system, and its delta algorithm buys nothing on compressed files that change wholesale. Manifest plus HTTP range requests gives the same two things that actually matter — skip what you have, resume what broke — over the connection you already trust.