Skip to content

08 · IPFS (Kubo)

SPIRENS runs Kubo, the reference Go implementation of IPFS, to host your own content-addressed storage and gateway. Once running, you get:

  • https://ipfs.example.com/ipfs/{cid} — path-style content access
  • https://{cid}.ipfs.example.com/ — subdomain-style content access (important for browser same-origin isolation between CIDs)
  • https://ipfs.example.com/ipns/{name} — path-style IPNS / DNSLink
  • https://{key}.ipns.example.com/ — subdomain-style IPNS (same origin-isolation benefit; routes via the *.ipns.$BASE wildcard)
  • A libp2p swarm port on :4001 TCP+UDP — peer connections

Why IPFS_PROFILE=server,pebbleds

The defaults in Kubo target a laptop; SPIRENS targets a server. Two profile tweaks make a material difference:

  • server — disables MDNS (LAN peer discovery, noisy on a datacenter network) and NAT port mapping (routers should forward 4001 explicitly, not through uPnP).
  • pebbleds — swaps the default LevelDB datastore for PebbleDB. Significantly faster at the steady-state write rate Kubo produces; CockroachDB uses the same engine in production.

Changing between flatfs, leveldb, and pebbleds after data exists requires a migration — pick one and stick with it.

Why the API is locked to loopback

/api/v0/* is Kubo's admin API. Anything on it can:

  • delete all your pinned content
  • change your node's identity
  • connect to arbitrary peers and consume your bandwidth
  • stream every file you own through to anyone

SPIRENS binds port 5001 to 127.0.0.1 only, and exposes it to other containers on the internal spirens_backend network via the Docker DNS name ipfs:5001. dweb-proxy is the only thing that reaches it. Do not publish port 5001 on the host even on a trusted LAN.

If you want to use the Kubo CLI from your workstation:

# SSH tunnel, not an exposed port
ssh -L 5001:127.0.0.1:5001 your-host
# then, on your workstation:
ipfs --api /ip4/127.0.0.1/tcp/5001 id

Subdomain gateway — why and how

A path-style gateway (ipfs.example.com/ipfs/{cid}) works but breaks browser security boundaries: two CIDs served from the same origin share cookies, localStorage, and service-worker scope. Malicious content A can read what content B stored.

The subdomain gateway fixes this by moving each CID to its own origin:

https://bafybei….ipfs.example.com/index.html

Each CID gets an isolated origin; same-origin policy does its job. Kubo handles the rewrite when you enable UseSubdomains: true on the gateway's public-gateway entry — spirens configure-ipfs does this for you.

Why wildcard DNS + wildcard TLS matter here: bafybei….ipfs.example.com is a new hostname per CID. You need *.ipfs.example.com in DNS (so it routes) and in the TLS cert (so browsers don't balk). Same applies to IPNS — {key}.ipns.example.com needs *.ipns.example.com wildcards.

All four are set up by SPIRENS out of the box:

  • DNS: see 02 — DNS & Cloudflare — the *.ipfs and *.ipns records.
  • TLS: see the tls.domains[0].sans=*.${IPFS_GATEWAY_HOST} entry (CID subdomain) and the parallel tls.domains[0].sans=*.ipns.${BASE_DOMAIN} entry (IPNS subdomain) in compose/single-host/compose.ipfs.yml. Two separate wildcard-cert requests; Traefik issues both at first boot.

Why path URLs don't auto-redirect to subdomains

Canonically, a Kubo gateway with UseSubdomains: true responds to GET /ipfs/{cid} with a 301 redirect to {cid}.ipfs.{gateway-host}. That's the "type into the URL bar once, land on an origin-isolated subdomain" flow most operators expect.

SPIRENS does not emit that redirect. Hitting https://ipfs.example.com/ipfs/{cid} serves the content directly on the path URL; it does not redirect to {cid}.ipfs.example.com. Both URLs work — they just don't redirect to each other.

Why. Kubo issue #9658: when the gateway host already contains ipfs. (like ipfs.example.com), UseSubdomains: true redirects to {cid}.ipfs.ipfs.example.com — a doubled ipfs. label, no cert, no route. Closed as won't-fix; the Kubo team treats "ipfs. at a sub-label" as operator-config, not a Kubo bug.

SPIRENS's workaround (see src/spirens/core/ipfs.py) is to register two Gateway.PublicGateways entries, matched by Host header:

  • ipfs.$BASE with UseSubdomains: false — path gateway, serves ipfs.example.com/ipfs/{cid} directly, never redirects.
  • $BASE (the apex, e.g. example.com) with UseSubdomains: true — subdomain gateway, recognises {cid}.ipfs.example.com and {key}.ipns.example.com and serves them directly.

No doubling because no entry has ipfs. in its own host label. You lose the canonical path→subdomain redirect, but both URL shapes work.

If you need the redirect flow, use a gateway host whose leftmost label is not ipfs. For example, setting IPFS_GATEWAY_HOST=gateway.example.com sidesteps the Kubo bug entirely — a single PublicGateways entry with UseSubdomains: true works as documented, and gateway.example.com/ipfs/{cid} redirects to {cid}.ipfs.gateway.example.com. That move also requires updating src/spirens/core/ipfs.py to emit one entry instead of two, plus new DNS/TLS wildcards (*.gateway.example.com). Not SPIRENS's default, but a clean path for anyone who needs the canonical redirect flow.

Post-deploy configuration

Some Kubo settings can only be set via the HTTP API after the node starts (CORS, gateway registration, DNS resolvers). SPIRENS applies them via spirens configure-ipfs, which spirens up runs automatically on first boot.

Re-run after container recreation:

spirens configure-ipfs

What it sets:

Key Value Why
API.HTTPHeaders.Access-Control-Allow-* ["*"] / [GET,POST,PUT] Browser dApps can call the API
Gateway.HTTPHeaders.Access-Control-Allow-* ["*"] / [GET,POST,PUT] Browser dApps can fetch content
Gateway.PublicGateways.{HOST}.UseSubdomains true Enables {cid}.ipfs.… rewrite
Gateway.PublicGateways.{HOST}.Paths ["/ipfs","/ipns"] Valid entry paths
Gateway.PublicGateways.{HOST}.NoDNSLink false DNSLink lookups enabled
DNS.Resolvers.eth. https://ens-resolver.…/dns-query .eth names resolve via dweb-proxy

Pinning content

Kubo will cache anything you fetch, but the cache gets GC'd periodically. pin add says "keep this forever":

docker exec spirens-ipfs ipfs pin add <cid>

For a managed pinning story (HA, redundancy across nodes, pinning API), look at ipfs-cluster. Out of scope for SPIRENS but the natural next step if your stack grows.

Peering for content availability

By default Kubo relies on the DHT for content discovery, which is best-effort. You can add explicit peering to well-known IPFS providers (Cloudflare, Filebase, 4EVERLAND, etc.) via the Peering.Peers config. That list is a moving target — SPIRENS doesn't ship one; check libp2p/specs/peering/ or your chosen provider's docs for their current peer IDs.

Gateway limits on cheap VPSes

The gateway streams content; big files hit bandwidth and memory. On a small VPS:

  • Set Datastore.StorageMax to something reasonable (default 10GB).
  • Watch RSS; Kubo can OOM with many concurrent pins. Add swap if you can't add RAM.
  • Consider Reprovider.Interval: "24h" (default is 12h) to reduce DHT chatter.

Bandwidth costs can surprise you

A public IPFS gateway serves whatever content anyone asks for (CIDs you host or fetch on demand). A single popular site pinned on your node can push hundreds of GB/month through your gateway. On cheap VPS plans with metered egress (Hetzner, DigitalOcean, Vultr), this is how a $5/month server becomes a $50/month server.

Mitigations, cheapest to most restrictive:

  1. Track usage early. docker exec spirens-ipfs ipfs stats bw and your provider's bandwidth graphs. Set an alert at 50% of your plan's monthly quota.
  2. Cloudflare proxy the gateway (ipfs A record orange-cloud). CF caches by URL, so repeat fetches of the same CID are free after the first hit. Note the 100 MB per-response cap on the Free plan.
  3. Rate-limit at Traefik. Add a rateLimit middleware to config/traefik/dynamic.yml and attach it to the IPFS gateway router — see 04 — Deployment Profiles: Rate limiting.
  4. IP-allowlist the gateway. If you're the only consumer, add dashboard-ipallow@file to the gateway's middlewares= label. You keep the gateway; random scrapers can't reach it.
  5. Serve only what you pin. Turn off DHT providing for non-pinned content with Routing.Type: autoclient (Kubo 0.21+). Your node becomes a consumer-only node for other CIDs; it only serves what you explicitly chose to host.

If you want a managed gateway and don't care about sovereignty, Cloudflare runs one at cloudflare-ipfs.com. SPIRENS's reason for existing is that you don't want someone else running it.

Continue → 09 — dweb-proxy