storage/gcp: optionally set object retention on immutable log resources - #1137
Open
eperrine-ant wants to merge 1 commit into
Open
storage/gcp: optionally set object retention on immutable log resources#1137eperrine-ant wants to merge 1 commit into
eperrine-ant wants to merge 1 commit into
Conversation
eperrine-ant
force-pushed
the
gcp-object-retention-hook
branch
from
August 21, 2026 16:41
bd35fbe to
c29b1f4
Compare
Add Config.ObjectRetentionPeriod and Config.ObjectRetentionLocked. When a period is set, the driver writes the log's immutable resources - full tiles and full entry bundles - with a GCS object retention configuration expiring that period after the write, in Locked or Unlocked mode, so that on a bucket with object retention enabled they cannot be overwritten or deleted until then. The checkpoint and partial tiles/bundles are always written without retention so that they can still be updated and garbage collected. Both the Appender and MigrationWriter lifecycles honour the option; left unset, behaviour is unchanged. The decision is made in logResourceStore, which knows from the layout which resources are partial, and passed to objStore.setObject as a new retention argument which gcsStorage sets on the Writer. Tests cover the layout decision (checkpoint and partials never retained, full tiles and bundles retained with the configured mode and expiry, and nothing retained when no period is set), that gcsStorage sends a non-nil retention as the object's retention metadata on upload and none for nil (against a minimal fake of the GCS JSON API), and validation of a negative period in New.
eperrine-ant
force-pushed
the
gcp-object-retention-hook
branch
from
August 21, 2026 16:43
c29b1f4 to
141468b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two optional fields to the GCP driver's
Config:ObjectRetentionPeriod time.Duration— when non-zero, the log's immutable resources (full tiles and full entry bundles) are written with a GCS object retention configuration expiring that period after the write. The checkpoint and partial tiles/bundles are always written without retention so they can still be updated and garbage-collected.ObjectRetentionLocked bool— selectsLocked(cannot later be reduced or removed) vsUnlockedmode for that retention.Left unset, behaviour is unchanged. Both
AppenderandMigrationWriterhonour it.Newrejects a negative period. (Object retention can only be set through the JSON API, which the default client uses; the field doc notes this for callers supplying their ownGCSClient.)Internally,
logResourceStoredecides from the layout (partial vs full) whether a resource gets retention, andobjStore.setObjectgains aretention *storage.ObjectRetentionargument whichgcsStoragesets on theWriter.Why
On a bucket with object retention enabled, this lets an operator make the log's write-once resources undeletable and unmodifiable (including by the log operator's own credentials) for a chosen period, as storage-level defence in depth for the log's append-only property. Doing it at write time avoids a second metadata-update round trip per object and the window in which the object exists unlocked.
A bucket-level retention policy cannot express this, because the checkpoint and
.p/partials must remain mutable/deletable — which is also why the driver, which owns the layout, is the right place to decide which objects are retained rather than the caller.Testing
TestImmutableResourceRetentiondriveslogResourceStoreover the in-memoryobjStoreand checks the checkpoint and partial tile/bundle are never retained, full tile/bundle are retained with the configured mode and an expiry of now + period, and nothing is retained when no period is set.TestSetObjectRetentiondrivesgcsStorageagainst anhttptestfake of the GCS JSON API and checks a non-nil retention arrives as the uploaded object'sretentionmetadata (mode,retainUntilTime) and a nil one sends none.TestObjectRetentionPeriodValidationcoversNew.go test ./storage/gcp/...,gofmt,golangci-lintclean.Alternative considered
A caller-supplied
func(objName string) *storage.ObjectRetentionhook evaluated per write. It is fewer lines in the driver and maximally flexible, but it makes the caller responsible for knowing which layout paths are safe to lock (get it wrong and checkpoint publication or GC breaks), so the driver-owned form above seemed the better default. Happy to go the other way if preferred.