feat: add point dimension tables and backfill jobs - #3375
Open
Freika wants to merge 2 commits into
Open
Conversation
Introduce point_sources and point_motions reference tables that deduplicate the device/importer combo columns and motion payloads repeated on every point. Both dedup through an md5 digest computed in SQL from canonical jsonb text, since motion payloads can exceed the btree row limit for a plain unique index. Backfill jobs walk points in id-range batches, seed the dimensions, stamp source_id/motion_id, and resolve country_id from country_name and the legacy country column. Both are resumable, idempotent, and enqueued automatically on upgrade with an env escape hatch. Also drops the visit-null partial index now that detection falls back to the consolidated unique index.
Bound the ALTER TABLE lock wait with retries and hand off to a dedicated job when the lock cannot be won, so boot never crash-loops on a busy points table. Make the enqueue migration the single decision point for starting the backfill chain, gate Cloud to manual scheduling, and own the retry lifecycle in the backfill jobs with bounded lock and statement timeouts. Stamp both dimensions in one UPDATE per batch instead of two, guard dimension inserts against burning sequence values on repeats, and chain the country backfill after the dimension walk finishes.
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
First slice of the points normalization (Release C of the storage plan; follows #3359/#3360/#3368). Adds the two dimension tables and the backfills — no read-path or ingest changes yet.
point_sources— deduplicates the nine device/importer combo columns (tracker_id,topic,ssid,bssid,connection,trigger,battery_status,inrids,in_regions). Measured on a 1.6M-point dataset these collapse to ~200 distinct rows.point_motions— deduplicatesmotion_datapayloads (~18× dedup measured).md5(jsonb::text), computed in SQL so every writer produces identical digests; jsonb output is canonical). A plain unique index on the jsonb is impossible — payloads can exceed the 2.7 kB btree row limit.pointsgains nullablesource_id/motion_id(int, metadata-only migration, instant). No FK constraints yet — they come with the rewrite.data_migrationsqueue, auto-enqueued on upgrade,SKIP_POINT_DIMENSION_BACKFILL=1to opt out): id-range batches of 50k with a 5s pause, self-re-enqueueing, resumable from any cursor, idempotent — seeding upserts by digest, stamping touches only NULL FKs. A second job resolvescountry_idfromcountry_name(42% of rows are NULL) plus the legacycountrycolumn, never overwriting an existing value.idx_points_user_visit_null_timestamp(visit detection falls back to the consolidated unique index; verified with EXPLAIN).Scoping
Ingest still writes only the legacy columns — the dual-write creators/importers and the read-path joins come in follow-up PRs before any legacy column is dropped. The backfill is re-runnable at any time to catch up rows created after its pass. Nothing user-visible changes.
Testing