refactor(materializer): remove the per-pass fold — the daily refresh is the only rollup writer (#55 step 5) - #65
Merged
Merged
Conversation
…is the only rollup writer (#55 step 5) With prod on MATERIALIZER_DAILY_ROLLUP_MODE=on, captureRollupDelta + foldSignalsRollup (and the _rollup_delta/_rollup_new temp tables) are dead code: delete them, so the span transaction is base-insert + late-subject marking only. The shadow-mode scaffolding that existed to validate the flip goes with them — DailyRollupShadow, DailyRollupDiff (and the diff-rows gauge), createDailyTable, the shadow branches of seedDailyRollup / observeRollupCardinality / PruneDecoded, and dailyTargetTable (always lake.signals_latest now). The boot-time promote of a leftover lake.signals_latest_daily table is KEPT: a node upgrading straight from shadow-era config still needs it, and after the promote the table is gone forever. ParseDailyRollupMode: empty now means ON — with the fold gone an unmaintained rollup must not be reachable by default; "shadow" is invalid so stale configs fail loud at boot; "off" survives for tests/one-off ops and warns at LoadDailyRollupState. FlushRollup's signals path serves only mode off + the backfill catch-up (its remaining consumers); RecomputeRollup keeps its mode-on refusal. dailyRollupSideRows only ever emits side="live"; the cardinality probe now also runs after every mode-on refresh (it previously rode on the shadow diff). LAKE_ROLLUP_DAILY_SERVING docs/sample flip to true — a plain rollup read under-counts the tail against a daily-maintained rollup. Tests: ducklake_incremental_rollup_test.go is deleted whole (it tested the deleted fold; its rollupRow/dumpRollupMap/incrRunner/drainNoFlush helpers move to rollup_helpers_test.go, joined by oracleRecompute and a refreshRollup helper that runs the production daily-refresh path). assertMatchesRecompute is deleted with it — the verify campaign now asserts through an explicit recompute, since the differential premise is gone. ducklake_daily_rollup_test.go is rewritten to prove seed/fold/late/ idempotency under mode on against lake.signals_latest with a full-recompute oracle; the shadow-diff "detects divergence" scenario is deleted (no diff exists to test). The flip test manufactures shadow-era leftovers directly and adds the aborted-seed drop path. Fold-dependent tests (rollup/query/kv/parity/retention/migration/only) gain an explicit refresh or recompute after draining; the subject-scoped flush and dirty-set-overflow tests switch to backfill mode, the machinery's remaining driver. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The final code step of #55: with prod on
MATERIALIZER_DAILY_ROLLUP_MODE=onfor three days (spans 275–515s → 6–19s, cardinality holding rows==keys through every refresh), the per-pass fold is dead code — and dead code that raced din's compaction into 823k duplicate rows when it was alive. This deletes it and retires the shadow scaffolding that existed to validate its replacement.What's deleted
captureRollupDelta+foldSignalsRollup+ their span-transaction call sites (the span txn is now base-insert + late-subject marking + KV publish + cursor, unconditionally).DailyRollupShadowmode, the shadow diff,dailyRollupDiffRows, the shadow-table creation path,PruneDecoded's shadow orphan block. Thelake.signals_latest_dailyname survives only in the boot-time promote (a node upgrading straight from shadow-era still gets the verified swap; an aborted shadow seed is dropped).tests/ducklake_incremental_rollup_test.gowholesale — every test in it existed to prove fold==recompute, and the fold is gone. Shared helpers moved totests/rollup_helpers_test.go; every surviving rollup assertion now pins against a recompute oracle or the production daily-refresh path (no assertions weakened; the commit body lists the per-test dispositions).Behavior changes
on(emptyMATERIALIZER_DAILY_ROLLUP_MODE⇒ daily refresh): with the fold gone, an unmaintained serving rollup must not be reachable by accident.offremains for tests and logs a warning;shadowno longer parses (configs must move toon— prod already has).LAKE_ROLLUP_DAILY_SERVINGsample/docs flipped totrueto match.kv_newernoise (the modes remain for gross-mismatch checks).Prod deploys this as a no-op (config already
on); the paired dimo-node PR will bump the image and returnMATERIALIZER_MAX_SNAPSHOT_SPANto 16 — the last line of #55.Includes #64's probe-per-refresh fix (rebased over its merge). Authored with a subagent executing a written spec; diff reviewed hunk-by-hunk and all gates re-run independently: build clean, full suite green, lint 0 issues.
Refs #55. Closes #55 when deployed.
🤖 Generated with Claude Code