Skip to content

storage/gcp: optionally set object retention on immutable log resources - #1137

Open
eperrine-ant wants to merge 1 commit into
transparency-dev:mainfrom
eperrine-ant:gcp-object-retention-hook
Open

storage/gcp: optionally set object retention on immutable log resources#1137
eperrine-ant wants to merge 1 commit into
transparency-dev:mainfrom
eperrine-ant:gcp-object-retention-hook

Conversation

@eperrine-ant

@eperrine-ant eperrine-ant commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What

Adds two optional fields to the GCP driver's Config:

  • ObjectRetentionPeriod time.Duration — when non-zero, the log's immutable resources (full tiles and full entry bundles) are written with a GCS object retention configuration expiring that period after the write. The checkpoint and partial tiles/bundles are always written without retention so they can still be updated and garbage-collected.
  • ObjectRetentionLocked bool — selects Locked (cannot later be reduced or removed) vs Unlocked mode for that retention.

Left unset, behaviour is unchanged. Both Appender and MigrationWriter honour it. New rejects a negative period. (Object retention can only be set through the JSON API, which the default client uses; the field doc notes this for callers supplying their own GCSClient.)

Internally, logResourceStore decides from the layout (partial vs full) whether a resource gets retention, and objStore.setObject gains a retention *storage.ObjectRetention argument which gcsStorage sets on the Writer.

Why

On a bucket with object retention enabled, this lets an operator make the log's write-once resources undeletable and unmodifiable (including by the log operator's own credentials) for a chosen period, as storage-level defence in depth for the log's append-only property. Doing it at write time avoids a second metadata-update round trip per object and the window in which the object exists unlocked.

A bucket-level retention policy cannot express this, because the checkpoint and .p/ partials must remain mutable/deletable — which is also why the driver, which owns the layout, is the right place to decide which objects are retained rather than the caller.

Testing

  • TestImmutableResourceRetention drives logResourceStore over the in-memory objStore and checks the checkpoint and partial tile/bundle are never retained, full tile/bundle are retained with the configured mode and an expiry of now + period, and nothing is retained when no period is set.
  • TestSetObjectRetention drives gcsStorage against an httptest fake of the GCS JSON API and checks a non-nil retention arrives as the uploaded object's retention metadata (mode, retainUntilTime) and a nil one sends none.
  • TestObjectRetentionPeriodValidation covers New.

go test ./storage/gcp/..., gofmt, golangci-lint clean.

Alternative considered

A caller-supplied func(objName string) *storage.ObjectRetention hook evaluated per write. It is fewer lines in the driver and maximally flexible, but it makes the caller responsible for knowing which layout paths are safe to lock (get it wrong and checkpoint publication or GC breaks), so the driver-owned form above seemed the better default. Happy to go the other way if preferred.

@eperrine-ant
eperrine-ant requested a review from a team as a code owner August 21, 2026 16:34
@eperrine-ant
eperrine-ant requested a review from roger2hk August 21, 2026 16:34
@eperrine-ant eperrine-ant changed the title storage/gcp: add optional per-object retention hook to Config storage/gcp: optionally set object retention on immutable log resources Aug 21, 2026
@eperrine-ant
eperrine-ant force-pushed the gcp-object-retention-hook branch from bd35fbe to c29b1f4 Compare August 21, 2026 16:41
Add Config.ObjectRetentionPeriod and Config.ObjectRetentionLocked. When a
period is set, the driver writes the log's immutable resources - full
tiles and full entry bundles - with a GCS object retention configuration
expiring that period after the write, in Locked or Unlocked mode, so that
on a bucket with object retention enabled they cannot be overwritten or
deleted until then. The checkpoint and partial tiles/bundles are always
written without retention so that they can still be updated and garbage
collected. Both the Appender and MigrationWriter lifecycles honour the
option; left unset, behaviour is unchanged.

The decision is made in logResourceStore, which knows from the layout
which resources are partial, and passed to objStore.setObject as a new
retention argument which gcsStorage sets on the Writer.

Tests cover the layout decision (checkpoint and partials never retained,
full tiles and bundles retained with the configured mode and expiry, and
nothing retained when no period is set), that gcsStorage sends a non-nil
retention as the object's retention metadata on upload and none for nil
(against a minimal fake of the GCS JSON API), and validation of a
negative period in New.
@eperrine-ant
eperrine-ant force-pushed the gcp-object-retention-hook branch from c29b1f4 to 141468b Compare August 21, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant