Skip to content

fix(pitr): cap checkpoint_timeout/max_wal_size during replay to bound disk use#106

Open
paulocsanz wants to merge 1 commit into
mainfrom
pc/pitr-recovery-checkpoint-cap
Open

fix(pitr): cap checkpoint_timeout/max_wal_size during replay to bound disk use#106
paulocsanz wants to merge 1 commit into
mainfrom
pc/pitr-recovery-checkpoint-cap

Conversation

@paulocsanz

Copy link
Copy Markdown
Contributor

Summary

  • During PITR replay, restore_command fetches WAL segments into pg_wal but Postgres only reclaims them at a restartpoint, whose frequency depends on whatever checkpoint_timeout/max_wal_size is already on disk (vanilla defaults, a customer's own prod tuning, or — on a volume cloned from the source — the source's own tuning).
  • On a volume provisioned near the source's current size, that buffer can push total disk usage past the base backup's footprint before the PITR target is reached, risking ENOSPC mid-restore.
  • Adds -c checkpoint_timeout=30s -c max_wal_size=512MB to the postgres invocation, but only for the boot that's actively replaying toward a PITR target (recovery.signal present + WAL_RECOVER_FROM_BUCKET set — covers both the empty-volume-restore path and the existing-volume replay path). Command-line flags win over postgresql.auto.conf regardless of what's already on disk, and apply to that one process only, so nothing needs to be reverted post-promote — the next boot just uses the Dockerfile CMD's plain args.
  • Base backup restore itself was already efficient (pgbackrest restore --delta writes directly into $PGDATA, no staging copy), so this closes the remaining disk-usage gap in the replay phase.

Test plan

  • bash -n wrapper.sh
  • ./test/e2e.sh t_empty_volume_restore_from_s3 — confirmed via docker exec ... ps aux that the running postgres process picks up -c checkpoint_timeout=30s -c max_wal_size=512MB only during the replay boot, and the restore/promote flow still passes
  • ./test/e2e.sh t_pitr_happy_path t_pitr_sentinel_blocks_retrigger t_restored_marker_persists_across_restarts t_restore_change_target_after_promote_noop t_chain_restore_r1_to_r2 t_pitr_retry_after_failed_staging t_restore_then_wipe_volume_redoes_restore t_restored_service_can_enable_archive_after_promote — all pass with the change
  • Compared against unmodified baseline (git stash) to confirm a couple of observed test failures (timing-sensitive setup_pitr_source target-capture races, documented in the test's own comments) reproduce identically without this change — pre-existing flakiness, not a regression

… disk use

Recovery.signal-driven WAL replay fetches segments into pg_wal but only
reclaims them at a restartpoint, whose cadence is set by whatever
checkpoint_timeout/max_wal_size is already on disk (vanilla defaults, a
customer's own prod tuning, or a cloned volume's inherited settings). On a
volume provisioned near the source's current size, that buffer risks
ENOSPC before the PITR target is reached. Pass -c overrides on the replay
boot's postgres invocation only — command-line wins over auto.conf
regardless of what's on disk, and self-reverts next boot with no cleanup
needed since it's process-scoped, not file-scoped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant