Skip to content

History cache-density knobs: decimation, incremental vacuum, blob compression#54

Merged
AndrewCTF merged 1 commit into
masterfrom
history-cache-density-config
Jul 18, 2026
Merged

History cache-density knobs: decimation, incremental vacuum, blob compression#54
AndrewCTF merged 1 commit into
masterfrom
history-cache-density-config

Conversation

@AndrewCTF

@AndrewCTF AndrewCTF commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

Makes the position-archive cache density tunable — store more replay per byte at
fixed disk. Three opt-in knobs, all OFF by default so the current on-disk
format and behaviour are unchanged. bash scripts/verify.sh green (1753 passed).

The knobs (Settings)

Setting What it does Lever
history_decimate_after_hours + history_decimate_stride Thin each track older than N hours to one fix in every stride during the maintenance pass. Recent data stays full-fidelity; the first fix of an old segment is always kept so short tracks don't vanish. Tiered resolution — old data goes coarse, so far more time-span fits the byte cap. Biggest density gain.
history_incremental_vacuum INCREMENTAL auto_vacuum + a cheap PRAGMA incremental_vacuum each pass instead of a periodic full VACUUM. Falls back to full VACUUM on an existing store until its next VACUUM converts it. Free-page reclaim — deleted pages return to the file continuously instead of accumulating between hourly passes.
history_compress_extra zlib-compress the per-fix extra JSON blob (alt/squawk/… metadata). The blob is write-only today — no query reads it — so there's no read path to change. Compression — a pure density win at write-CPU cost.

_encode_extra / _decode_extra handle the (de)compression and tolerate a mixed
column, so the flag can be flipped without a migration and a future reader has a
decode path.

Not included (deliberately)

  • Coordinate quantization (lat/lon/track REAL → scaled INT) is the other
    big density lever, but it's a schema migration over the whole archive, not a
    runtime flag — a follow-up with its own migration, not a dead config knob here.
  • Value-based eviction — for a uniform time-series the value metric is
    recency, which the existing time-window + byte-cap retention already implements.
    Decimation is the density knob; no separate setting needed.

Tradeoffs

Decimation loses old-data precision; compression and incremental vacuum add CPU.
That's the intended trade — density for compute/fidelity, not disk. All default
off, so nothing changes until an operator opts in.

Tests

  • test_decimate_thins_old_tracks_keeps_recent — exact-count thinning, recent
    data untouched, and a short old track keeps its first fix (falsified: dropping
    the first-fix rule makes the short track vanish and the test fails).
  • test_incremental_vacuum_mode_and_reclaim — a fresh DB comes up in INCREMENTAL
    mode and _reclaim runs without error.
  • test_encode_decode_extra_roundtrip — both modes round-trip, and decode
    tolerates a mixed column and garbage.

Store more replay per byte at fixed disk, all OFF by default so the current
on-disk format and behaviour are unchanged:

- history_decimate_after_hours / _stride: tiered resolution — thin each track
  older than N hours to one fix in every stride during the maintenance pass.
  Recent data stays full-fidelity and the first fix of an old segment is always
  kept, so short tracks don't vanish; old data becomes coarse, so far more
  time-span fits under the same byte cap.
- history_incremental_vacuum: INCREMENTAL auto_vacuum plus a cheap PRAGMA
  incremental_vacuum each maintenance pass instead of a periodic full VACUUM, so
  deleted pages return to the file continuously (falls back to full VACUUM on an
  existing store until its next VACUUM converts it).
- history_compress_extra: zlib-compress the per-fix extra JSON blob, which is
  write-only today (no query reads it), via _encode_extra / _decode_extra.

Coordinate quantization (REAL -> scaled INT) is a schema migration rather than a
runtime flag, so it's left as a follow-up; value-based eviction is already the
recency-based retention policy.
@AndrewCTF
AndrewCTF merged commit dd30aad into master Jul 18, 2026
2 checks passed
@AndrewCTF
AndrewCTF deleted the history-cache-density-config branch July 18, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant