🐝 DeHive

The universal data bank format — pack a directory into one signed, deduplicated, seekable blob and stop shipping 40 000 loose files.

---

🤔 What is this?

DeHive (.hb) is a binary data bank format + tooling for stuffing a whole folder of assets into one file — with per-entry CRC64, optional compression, dedup, obfuscation, RSA signing, and a table-of-contents footer that lets you open a 50 GB bank without reading 50 GB.

"Couldn't I just use a .zip?"

You could. And your game would open a fresh file handle per texture, seek all over the archive on boot, re-hash the world to verify integrity, and ship a 4 GB patch because one .png changed and shifted every byte after it. But sure. .zip. 🙂

DeHive is built for runtime asset loading — think game engines — where you care about open speed, random access, streaming, patching, and not trusting the disk.

---

✨ Features

	Feature	What it buys you
📇	TOC footer	O(1) open — no sequential header scan across the whole file
🎯	RandomAccess reads	One shared file handle per bank, positionless reads, thread-safe. Zero FileStream-per-asset churn
🧠	Zero-copy mmap	TryGetMemory hands stored assets straight from mapped pages as ReadOnlyMemory<byte> — no buffer copy, ideal for GPU upload
🪟	Seekable chunks	Large compressed assets split into independently-decompressed blocks — stream the middle of an asset without inflating the whole thing
📚	Batch loading	ReadManyAsync dispatches reads in physical-offset order with bounded concurrency — near-sequential IO instead of random seeks
🧹	Load filtering	.hiveignore (gitignore syntax) + include/exclude globs — junk (.DS_Store, Thumbs.db, .git/) skipped by default
🧩	Master + shards	Sharded builds emit a tiny master bank (index only) — boot touches one small file, data shards open lazily
✂️	Patch-stable shards	Content-defined boundaries + content-derived shard names — adding one asset reshuffles ~1 shard, not all of them, so per-file delta patches stay tiny
📐	Block alignment	Align entry data to N bytes (mmap / DirectStorage / smaller binary deltas)
🗜️	Pluggable codecs	Per-entry Zstd (default) / Brotli / Store. Zstd decompresses multi-GB/s (Brotli doesn't); already-compressed assets (png/ogg/mp4) auto-stored
♻️	Content dedup	Identical files stored once, referenced by the rest
🔒	RSA-2048 signing	Tamper detection — verification refuses a modified bank
🥸	Scramble	Per-entry XOR keystream obfuscation (seekable) — keep casual data-miners out
🧮	CRC64 per entry	Cheap, lazy integrity checks — verify one asset on load, not the whole bank
🧬	Deterministic builds	Same input → bit-for-bit identical output. Delta patchers rejoice
🧅	Layered mounts	HiveBankGroup — DLC/patch banks override the base (last mounted wins)
🌐	IFileProvider	Drop a bank straight into ASP.NET static files / config
⚡	NativeAOT	The hive CLI ships as a single self-contained native binary

---

📦 Install

Library (NuGet):

dotnet add package DeHive

CLI — as a .NET tool:

dotnet tool install -g DeHive.Cli
hive --help

CLI — as a standalone NativeAOT binary (no runtime required):

OS	Download
🪟 Windows x64	hive-win-x64.zip → hive.exe
🐧 Linux x64	hive-linux-x64.tar.gz → hive

Grab it from Releases, unzip, done. One file. No apt install fourteen-things.

---

🚀 Quick start (CLI)

# Pack ./assets into game.hb — compressed, deduplicated, reproducible
hive pack game -f ./assets -o ./dist --compress --dedup --deterministic

# Split into shards + a master index bank (great for big projects & patching)
hive pack game -f ./assets -o ./dist --sharding --shard-size 512 --align 4096

# Inspect what actually made it into the build
hive view ./dist/game.hb

# Trust, but verify (CRC64 per entry)
hive verify ./dist/game.hb

# Get it all back out
hive extract ./dist/game.hb ./unpacked

hive view — because "why is my build 8 GB" deserves an answer.

---

🧑‍💻 Library usage

Open & read

using DeHive;

// HiveBank.OpenAsync auto-detects: plain data bank vs. sharded master bank
await using var bank = await HiveBank.OpenAsync(new FileInfo("game.hb"));

// Look up by path (case-insensitive, separator-agnostic — artists gonna artist)
await using var texture = bank.FindFile(new HiveRelativePath("textures/hero.png"));

// ...or by stable entity id (the fast path for a runtime)
await using var stream = bank.FindFile(entityId);

Zero-copy & batch loading

// Zero-copy: stored (uncompressed) assets come straight from mapped pages — no buffer copy.
if (bank.TryGetMemory(new HiveRelativePath("meshes/rock.bin"), out ReadOnlyMemory<byte> mem))
    UploadToGpu(mem.Span);                 // valid until the bank is disposed

// Bulk load: reads dispatched in physical-offset order with bounded concurrency.
var assets = await bank.ReadManyAsync(level.AssetPaths, maxConcurrency: 8);

// Seekable streaming: with --chunk-size, only the chunk holding the offset is decompressed.
await using var s = bank.FindFile(new HiveRelativePath("audio/music.ogg"));
s.Seek(2_000_000, SeekOrigin.Begin);       // jump into a compressed asset, decode just that block

Verify & extract

foreach (var r in await bank.VerifyAllAsync())
    if (!r.IsValid) Console.WriteLine($"💀 corrupt: {r.RelativePath}");

await bank.ExtractAllAsync(new DirectoryInfo("./unpacked"));

Signed banks

using var pub = RSA.Create();
pub.ImportSubjectPublicKeyInfo(publicKeyBytes, out _);

// Throws HiveSignatureException if the bank was tampered with
await using var bank = await HiveDataBank.OpenAsync(file, verificationKey: pub);

DLC / patches — layered mounts

// Later banks override earlier ones. base ➜ dlc ➜ hotfix.
await using var group = await HiveBankGroup.OpenAsync(
    new FileInfo("base.hb"),
    new FileInfo("dlc.hb"),
    new FileInfo("hotfix.hb"));

// hotfix.hb wins if it has this file, else dlc.hb, else base.hb
await using var s = group.FindFile(new HiveRelativePath("levels/01.map"));

As an IFileProvider

var provider = new HiveBankFileProvider(bank);
app.UseStaticFiles(new StaticFileOptions { FileProvider = provider });

Build one in code

await HiveDataBank.CreateAsync(
    new HiveDatabankSettings(root, output, shardMaximumSize: 512 * 1024 * 1024, blockAlign: 4096, "game")
    {
        UseCompression = true,
        EnableDeduplication = true,
        Deterministic = true,
    },
    builder => builder.AddFolder(root));

---

🎮 Why a game engine cares

DeHive was shaped for runtime asset loading, and it shows:

• 🏎️ Fast boot — TOC footer + master bank means opening a sharded, multi-gigabyte asset set touches a tiny index, not the whole thing.
• 🧵 Concurrent streaming — one SafeFileHandle per bank, positionless RandomAccess reads. Load a hundred assets across threads without a single stream-position mutex fight.
• 📐 mmap / DirectStorage-friendly — block alignment keeps entry data on clean boundaries.
• 🩹 Cheap patches — deterministic, aligned output + content-defined shard boundaries: adding or resizing one asset only rewrites the shard(s) near it, every other shard file keeps the same name and bytes. A per-file delta patcher then sees a handful of changed files instead of "everything moved." (Measured: inserting one asset left 10 of 13 shard files byte-identical; fixed positional shards left 0 of 7.)
• 🧅 DLC & mods for free — HiveBankGroup is literally "mount another bank, it wins." Base game, DLC, community mods — same mechanic.
• 🗜️ The right codec — Zstd by default (decompresses several times faster than Brotli, which is what gates load times), Store for already-compressed assets so you don't burn CPU re-squeezing a PNG. Brotli's still there when size beats speed.
• 🕵️ Lazy integrity — CRC64 per entry lets you validate an asset when you load it, instead of re-hashing the planet on startup.

---

🛠️ CLI reference

Command	Does
hive pack <name> -f <dir> -o <out>	Pack a folder into a bank
hive publish <manifest.json>	Pack from a manifest file
hive view <bank.hb>	Header, flags, shard list & entity table
hive verify <bank.hb>	CRC64 integrity check, every entry
hive extract <bank.hb> [out]	Unpack a bank to a directory
hive cat <bank.hb> <entry> [-o file]	Dump one entry's bytes to stdout or a file
hive cert <manifest.json>	Generate an RSA signing key pair
hive s3 upload ...	Push a bank to S3-compatible storage

pack flags

Flag	Effect
--compress, -c	Compress entries with the default codec (Zstd)
--codec <zstd|brotli|store>	Pick the codec (implies --compress unless store). Zstd = fast decompress, Brotli = smallest, Store = none
--compression-level <fastest|optimal|smallest>	Trade build time for size (decompression is unaffected)
--chunk-size <KB>	Split large compressed entries into N-KB seekable chunks (0 = whole-entity)
--exclude <glob> / --include <glob>	Filter files (gitignore syntax, repeatable). Combined with .hiveignore
--dedup	Store identical files once
--scramble	XOR-obfuscate entry data
--sign <cert.pfx>	RSA-sign the bank
--align <N>	Align entry data to N-byte boundaries
--sharding	Split into shards + emit a master index bank
--shard-size <MB>	Max shard size (with --sharding)
--fixed-shards	Disable content-defined boundaries — positional {bank}.NNNN.hb cut at max size (worse for patching)
--deterministic	Reproducible, bit-for-bit output
--json	Machine-readable JSONL output (see below)
--verbose	Full stack traces when things go sideways

Manifest (hive publish)

{
  "name": "game",
  "root": "./assets",
  "output": "./dist",
  "compression": true,
  "codec": "zstd",
  "compressionLevel": "optimal",
  "deduplication": true,
  "scramble": false,
  "sharding": true,
  "shardSizeMb": 512,
  "align": 4096,
  "deterministic": true,
  "certificates": { "PrivateCert": "./game.pfx", "PublicCert": "./game.pub" }
}

---

🤖 Machine-readable output (--json)

Add --json to any command and the CLI speaks JSONL — one JSON object per line on stdout — while every human-facing panel, table and progress bar is shunted to stderr. This is how you wire hive into a toolchain (say, a game-engine editor spawning it as a subprocess): read stdout line by line, surface log/error events, consume the final result. Exit code still means what you'd expect.

hive pack game -f ./assets -o ./dist --compress --sharding --json

{"task":"pack","current":8,"total":17,"kind":"progress"}
{"task":"pack","current":17,"total":17,"kind":"progress"}
{"bank":"game","entities":17,"files":[{"name":"game.0000.hb","size":821696},{"name":"game.hb","size":1361}],"totalBytes":823057,"elapsedMs":65,"sharded":true,"master":true,"kind":"result","command":"pack"}

Event kinds (discriminated by kind):

kind	When	Shape
progress	Long ops (pack/publish/extract), one per whole-percent	{ task, current, total }
result	Terminal success payload, command-specific	pack / verify / view / extract — see below
error	Anything that went wrong (also sets a non-zero exit code)	{ message, type, stack? }
log	Diagnostic fragments	{ level, message }

// verify — per-entry integrity + summary
{"total":17,"passed":17,"failed":0,"entries":[
  {"path":"textures/hero.png","expectedCrc":"0x69AB...","actualCrc":"0x69AB...","valid":true}
],"elapsedMs":18,"kind":"result","command":"verify"}

// error — machine-parseable, no ANSI, no guessing
{"message":"File is not hive bank (header is not valid)","type":"DeHive.Abstractions.Hives.HiveAssertException","stack":null,"kind":"error"}

Pretty output on stderr, data on stdout, exit code for the verdict. Your editor's error panel finally gets to stop scraping ANSI escape codes. 🎛️

---

🧱 Format at a glance

┌─────────────────────────────────────────────────────────┐
│ HiveBankHeader   magic "HIVE.BANK", v3, flags, align,    │
│                  shard id, entity/component/archetype cnt │
├─────────────────────────────────────────────────────────┤
│ Entity[0]  header + ".e_file" + (aligned) data           │
│ Entity[1]  ...                                            │
│   ...      (compressed? scrambled? all per-entry)         │
├─────────────────────────────────────────────────────────┤
│ TOC footer  path → offset/length/crc64/id/flags   ◄──────┼── opened first, O(1)
│ tocOffset + "TOC_END!"                                    │
└─────────────────────────────────────────────────────────┘

A MASTER bank holds no entity data — just the shard list and a global
path → (shard, offset) index. Open one file, stream from many.

🧬 Deterministic builds

With --deterministic, file ordering and entity IDs are derived from content (CRC64) and normalized paths — no timestamps, no random ULIDs. Same input, on any machine, produces byte-identical output. Which means your diff tool, your CI cache, and your patch generator all see "nothing changed" instead of "everything changed because the clock ticked."

✂️ Patch-stable shards (content-defined boundaries)

Sharding splits a big bank into several .hb files. The obvious way — "start a new shard every N megabytes" — has a nasty property for patching: the boundary is tied to absolute position. Add or resize one asset near the front and every later shard shifts, so a per-file delta patcher (the kind that pairs old/new by filename and diffs) sees every shard as changed and re-downloads the lot.

fixed size cut:                       content-defined cut:
  add one early asset ↓                  add one early asset ↓
  [S0][S1][S2][S3][S4]                   [S0][S1][S2][S3][S4]
  [S0'][S1'][S2'][S3'][S4']  ← all move  [S0'][S1][S2][S3][S4]  ← only S0 area moves
   every shard renamed+rewritten          the rest keep name + bytes

DeHive cuts shards the way backup tools (restic/borg) chunk data — the boundary follows content, not position:

• Boundary — between files (sorted by path), the cut point is chosen by a hash of the file path within a [¼·max, max] size band. Inserting a file only perturbs the shard around it; downstream shards re-sync to identical membership.
• Name & IDs — each shard is named {bank}.{hash-of-its-members}.hb (not a positional 0000), and its header IDs come from the same hash. So a shard whose members didn't change keeps the same filename and the same bytes across builds — exactly what a per-file patcher needs to skip it.

Measured — pack 40 assets sharded, then insert one asset that sorts first and re-pack:

	shards	survived byte-identical
content-defined (default)	13	10 / 13 ✅
fixed (--fixed-shards)	7	0 / 7 💀

So adding one asset patches ~3 small shard files + the tiny master, instead of the whole set.

Trade-off: content-defined shards average ~0.6× the max size (hence 13 vs 7 above) — a few more, smaller files in exchange for patch stability. The --shard-size hard cap is still never exceeded. Pairs with --deterministic (which you want anyway for reproducible patches); use --fixed-shards to opt back into positional numbering.

This is shard-level chunking (whole files). Sub-file chunking — splitting a single huge asset into seekable blocks so you can stream its middle without inflating the whole thing — is a separate feature: pack with --chunk-size.

---

🏗️ Building from source

git clone https://github.com/argon-chat/DeHive.git
cd DeHive/src/dotnet

dotnet build -c Release                            # build everything
dotnet test                                        # 31 tests, all green ✅
dotnet publish DeHive.Cli -c Release -r win-x64    # NativeAOT binary

Requirements: .NET 10 SDK. For NativeAOT on Linux: clang + zlib1g-dev. On Windows: the VS C++ build tools (you probably already have them).

---

📄 License

MIT © 2024-2026 Yuuki Wesp / Argon Inc. LLC / argon-chat

Made with 🐝 and an unreasonable dislike of loose files.