History cache-density knobs: decimation, incremental vacuum, blob compression#54
Merged
Merged
Conversation
Store more replay per byte at fixed disk, all OFF by default so the current on-disk format and behaviour are unchanged: - history_decimate_after_hours / _stride: tiered resolution — thin each track older than N hours to one fix in every stride during the maintenance pass. Recent data stays full-fidelity and the first fix of an old segment is always kept, so short tracks don't vanish; old data becomes coarse, so far more time-span fits under the same byte cap. - history_incremental_vacuum: INCREMENTAL auto_vacuum plus a cheap PRAGMA incremental_vacuum each maintenance pass instead of a periodic full VACUUM, so deleted pages return to the file continuously (falls back to full VACUUM on an existing store until its next VACUUM converts it). - history_compress_extra: zlib-compress the per-fix extra JSON blob, which is write-only today (no query reads it), via _encode_extra / _decode_extra. Coordinate quantization (REAL -> scaled INT) is a schema migration rather than a runtime flag, so it's left as a follow-up; value-based eviction is already the recency-based retention policy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the position-archive cache density tunable — store more replay per byte at
fixed disk. Three opt-in knobs, all OFF by default so the current on-disk
format and behaviour are unchanged.
bash scripts/verify.shgreen (1753 passed).The knobs (
Settings)history_decimate_after_hours+history_decimate_stridestrideduring the maintenance pass. Recent data stays full-fidelity; the first fix of an old segment is always kept so short tracks don't vanish.history_incremental_vacuumauto_vacuum+ a cheapPRAGMA incremental_vacuumeach pass instead of a periodic fullVACUUM. Falls back to fullVACUUMon an existing store until its nextVACUUMconverts it.history_compress_extraextraJSON blob (alt/squawk/… metadata). The blob is write-only today — no query reads it — so there's no read path to change._encode_extra/_decode_extrahandle the (de)compression and tolerate a mixedcolumn, so the flag can be flipped without a migration and a future reader has a
decode path.
Not included (deliberately)
REAL→ scaledINT) is the otherbig density lever, but it's a schema migration over the whole archive, not a
runtime flag — a follow-up with its own migration, not a dead config knob here.
recency, which the existing time-window + byte-cap retention already implements.
Decimation is the density knob; no separate setting needed.
Tradeoffs
Decimation loses old-data precision; compression and incremental vacuum add CPU.
That's the intended trade — density for compute/fidelity, not disk. All default
off, so nothing changes until an operator opts in.
Tests
test_decimate_thins_old_tracks_keeps_recent— exact-count thinning, recentdata untouched, and a short old track keeps its first fix (falsified: dropping
the first-fix rule makes the short track vanish and the test fails).
test_incremental_vacuum_mode_and_reclaim— a fresh DB comes up in INCREMENTALmode and
_reclaimruns without error.test_encode_decode_extra_roundtrip— both modes round-trip, and decodetolerates a mixed column and garbage.