fix(pitr): cap checkpoint_timeout/max_wal_size during replay to bound disk use#106
Open
paulocsanz wants to merge 1 commit into
Open
fix(pitr): cap checkpoint_timeout/max_wal_size during replay to bound disk use#106paulocsanz wants to merge 1 commit into
paulocsanz wants to merge 1 commit into
Conversation
… disk use Recovery.signal-driven WAL replay fetches segments into pg_wal but only reclaims them at a restartpoint, whose cadence is set by whatever checkpoint_timeout/max_wal_size is already on disk (vanilla defaults, a customer's own prod tuning, or a cloned volume's inherited settings). On a volume provisioned near the source's current size, that buffer risks ENOSPC before the PITR target is reached. Pass -c overrides on the replay boot's postgres invocation only — command-line wins over auto.conf regardless of what's on disk, and self-reverts next boot with no cleanup needed since it's process-scoped, not file-scoped.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
restore_commandfetches WAL segments intopg_walbut Postgres only reclaims them at a restartpoint, whose frequency depends on whatevercheckpoint_timeout/max_wal_sizeis already on disk (vanilla defaults, a customer's own prod tuning, or — on a volume cloned from the source — the source's own tuning).-c checkpoint_timeout=30s -c max_wal_size=512MBto the postgres invocation, but only for the boot that's actively replaying toward a PITR target (recovery.signalpresent +WAL_RECOVER_FROM_BUCKETset — covers both the empty-volume-restore path and the existing-volume replay path). Command-line flags win overpostgresql.auto.confregardless of what's already on disk, and apply to that one process only, so nothing needs to be reverted post-promote — the next boot just uses the Dockerfile CMD's plain args.pgbackrest restore --deltawrites directly into$PGDATA, no staging copy), so this closes the remaining disk-usage gap in the replay phase.Test plan
bash -n wrapper.sh./test/e2e.sh t_empty_volume_restore_from_s3— confirmed viadocker exec ... ps auxthat the running postgres process picks up-c checkpoint_timeout=30s -c max_wal_size=512MBonly during the replay boot, and the restore/promote flow still passes./test/e2e.sh t_pitr_happy_path t_pitr_sentinel_blocks_retrigger t_restored_marker_persists_across_restarts t_restore_change_target_after_promote_noop t_chain_restore_r1_to_r2 t_pitr_retry_after_failed_staging t_restore_then_wipe_volume_redoes_restore t_restored_service_can_enable_archive_after_promote— all pass with the changegit stash) to confirm a couple of observed test failures (timing-sensitivesetup_pitr_sourcetarget-capture races, documented in the test's own comments) reproduce identically without this change — pre-existing flakiness, not a regression