Pombo Storage

A single-page, backend-less client for uploading and downloading files directly to/from the Streamr storage-node + Cassandra cluster that backs Pombo.

Stack

Layer	Tech	Purpose
UI	HTML + Tailwind + FontAwesome	single-page, no build step
Network	@streamr/sdk 103.0.0	Streamr Network overlay (WebRTC data channels, WS fallback); publishes are signed with an Ethereum private key
Compression	pako 2.1.0 (DEFLATE, level 1)	applied selectively by file type (see Upload)
Video preview	mp4box.js 0.5.2	remuxes MP4→fMP4 into a MediaSource for progressive playback while still downloading
Local staging	OPFS via a dedicated Web Worker	avoids holding the whole file in memory; falls back to in-memory if OPFS is unavailable

Backend: Streamr cluster + Cassandra

• 2 VPS nodes, each running a stock Streamr storage node (@streamr/node v103.3.1) and a local Cassandra 4.1.11 instance.
• Both nodes share one on-chain storage-node identity, registered with two independent HTTPS URLs — there's no load balancer; it's redundancy via a shared identity with two independently reachable backends. Uploads verify/write against the primary URL; downloads round-robin reads across both.
• Cassandra: NetworkTopologyStrategy{dc1:2} (RF=2, full copy on each node), no auth (private-network only), writes at LOCAL_ONE consistency. Two tables: bucket (metadata) and stream_data (payload).
• Custom patches on the storage nodes (lost on @streamr/node upgrades):
  • maxBucketSize reduced 100MB→32MB and the full-bucket check interval to 250ms, to avoid oversized partitions under fast uploads.
  • A new ?format=metadata query param on the resend endpoints, returning only {timestamp, sequenceNumber, publisherId, size} with no payload — this is what makes this app's chunk-verification ~3400x cheaper than a full read. Additive: unpatched nodes return HTTP 400 and the client silently falls back to full reads.

Stream layout

• Partition 0 — announcements (file_announcement): name, type, sizes, compression, chunk count, transferId, timestamps, started/complete status.
• Partitions 1–9 — chunk data, round-robin (chunkPartition(i) = 1 + (i % 9)) to keep storage-node buckets small and allow parallel per-partition resends.
• Each chunk payload: [4B metadata length][metadata JSON][4B total chunks][4B chunk index][chunk bytes], published as raw binary.
• Chunk size fixed at 240KB: the Streamr wire protocol allows up to 1MiB node-to-node, but browser publishes go over WebRTC/SCTP whose negotiated maxMessageSize is typically 256KiB.

Upload flow (uploadFile())

1. Compression decision: video/audio, already-compressed images, and archives are sent raw (DEFLATE gains ~0% and blocks progressive preview). Everything else is deflated with pako.
2. MP4 faststart: a pure-JS qt-faststart reimplementation moves the moov atom before mdat and patches the offset tables, so playback can start before the download finishes.
3. Streaming compression: compressible files are deflated in a Worker straight to OPFS, so chunk publishing can start before the whole file is compressed.
4. Partition warm-up: a 1-byte ping to each of the 9 data partitions, then a 2s wait — each partition's overlay only forms on first publish, and anything sent before neighbors join is silently lost.
5. Chunk publish with retry and real backpressure (see below) — progress is tracked from the data channels' actual outbound queue, not the SDK's "sent" count.
6. Verify & repair (on by default): confirms each chunk actually landed on the cluster via ?format=metadata, waits for each partition to catch up, and re-publishes anything still missing (up to 8 passes).
7. Final announcement (status: 'complete', with the confirmed chunk count), published and re-confirmed with retries.

Download flow (downloadFile())

1. List available files by resending the last 50 messages on partition 0.
2. Compute the time window and per-partition ranges to fetch from the announcement's timestamps.
3. Parallel range reads, round-robined across both storage-node URLs, streamed and JSON-parsed incrementally rather than waiting for the full response body.
4. Resume support: received-chunk checkpoints are persisted periodically so an interrupted download can continue instead of restarting.
5. OPFS write-back with its own backpressure (writes land at absolute offsets as chunks arrive, out of order is fine; pending-write bytes are capped).
6. Progressive preview: for uncompressed video/audio, the already-written prefix is exposed as a growing Blob (fed through mp4box.js + MediaSource for MP4) while the rest keeps downloading.
7. Finalize: once all chunks arrive, compressed files are inflated (streamed through a Worker) and the final Blob is handed to the browser as a download.

Network resilience & auto-tuning

• Real-queue radar: hooks RTCPeerConnection/WebSocket send paths to read the actual outbound bufferedAmount, since the SDK's own "sent" count resolves before the data is actually on the wire.
• Hard backpressure: all sends pause once the outbound queue exceeds 1.5MB, resuming only below 384KB, to prevent silent SCTP overflow drops.
• Auto-tune: dynamically adjusts the delay between publishes based on observed saturation, converging on sustainable throughput without manual configuration.