feat: deferred-accept multipart (FIL-520) + network blob removal on DeleteObject (FIL-588)

[frrist]

Two features riding the blob-removal chain, aligned with the blob-removal RFC (fil-one/RFC#13) and merged onto main's Hilt architecture:

• FIL-520 — multipart, completed: ListParts, ListMultipartUploads, S3 validation, abort/expiry hygiene; parts park on piri at UploadPart (allocated + PUT, not accepted), Complete concludes them into acceptance, Abort unwinds them via /blob/abort → /blob/reject. A node's BlobAccepted refusal is treated as final (a concurrent session in the space accepted the same content — the reference index owns the blob), not a retryable fault.
• FIL-588 — DeleteObject releases network blobs: the last reference to a blob triggers /blob/remove (space-subject, {digest} args) through the upload service, which forwards /blob/release to the nodes; bytes are released by piri's async removal sweep.

Merged with main's Hilt refactor (per-bucket spaces, request-scoped proofs, fil-forge/versitygw fork): the deferred seams (UploadBlobParked/ConcludeBlob/AbortBlob) and RemoveBlob take the bucket's space; park/upload capture the request's per-key proof store as the space's write authority, and abort/remove resolve request-first with that captured store as the off-request fallback (the session-expiry sweeper). seedSpaceDelegations stays deleted — space authority is Hilt-issued per access key.

Open dependencies

• hilt: s3perm.go delegates no blob.Abort at all today, so every /blob/abort (client Abort, the session-expiry sweeper, DeleteBucket's implicit abort) goes out proofless and parked part blobs leak on abort. feat(s3perm): grant blob.Abort with the write command set hilt#36 adds blob.Abort to the write command set; CompleteMultipartUpload_conditional_writes and the AbortRejects gate promote once a hilt image with it publishes.
• smelt: feat(stack): delegate blob/release + blob/reject from piri nodes to upload smelt#19 (the /blob/release node delegation) sits on a pre-Hilt base and must rebase onto smelt main; this branch pins main's Hilt-era smelt until then, so the two forge itests below can't pass end-to-end yet.

Landing order

fil-forge/libforge#49 (client-verb args + named errors) → fil-forge/piri#30 (/blob/release + /blob/reject) → fil-forge/sprue#33 + fil-forge/smelt#19 (rebased) + the hilt blob.Abort grant → this. Re-pin libforge/smelt to merge commits as they land.

Validated: full unit suite, the S3 conformance partition (219 pass / 0 fail locally), gofmt. The end-to-end gates — TestForgeDeleteReleasesNetworkBlob (delete-finality through piri's removal sweep) and TestForgeDeferredMultipart (park/conclude round-trip + abort→reject) — now run on the stock smelt-SDK images like every other itest, and both pass locally against branch-built piri#30 and hilt#36 images (the full chain: ingot → sprue#33 → piri /blob/release+/blob/reject, delegations from smelt#19). They go green in CI once piri#30 and hilt#36 publish to :main; until then INGOT_ITEST_PIRI_IMAGE / INGOT_ITEST_HILT_IMAGE cover local runs.

🤖 Generated with Claude Code

[alanshaw]

Do we need to defer accept? By not accepting the Piri operator is storing data and not getting paid. There's also an attack vector here where users upload many multipart parts and the server runs out of space...they have no incentive to not do this as they will not be charged until they complete/abort. This is at odds with the AWS S3 implementation which charges you for storage of multipart parts until they are aborted or the session expires (which IIRC by default it does not).

[alanshaw]

sealed?

Suggested change

	// PutInvocation is the sealed /http/put invocation. Its metadata embeds
	// PutInvocation is the issued /http/put invocation. Its metadata embeds

[alanshaw]

I'm not a big fan of the park metaphor. It was kinda relevant//cute/fun when we were uploading CAR files but I'm not sure this holds now.

Could we instead just add a option to BlobAdd e.g. WithConclude(false). We add the fields from ParkedBlob to AddedBlob and let Location be nil when you don't conclude?

Then we just expose BlobConclude to allow later conclusions...

[alanshaw]

The /space prefix was dropped when we transitioned to UCAN 1.0.

Suggested change

	add_task bytea NOT NULL, -- /space/blob/add task CID (unallocate cause)
	add_task bytea NOT NULL, -- /blob/add task CID (unallocate cause)

[alanshaw]

UCANs are not sealed.

Suggested change

	put_invocation bytea NOT NULL, -- sealed /http/put invocation
	put_invocation bytea NOT NULL, -- issued /http/put invocation

[alanshaw]

Not sure there's any getting away from this...park in ingrained. What about multipart_blobs or multipart_added_blobs?

[alanshaw]

Suggested change

	type BlobPark struct {
	type MultipartAddedBlob struct {

or

Suggested change

	type BlobPark struct {
	type MultipartBlob struct {

[alanshaw]

I think "parks" is too obtuse.

Suggested change

	parks map[string]registry.BlobPark // keyed by string(digest)
	blobAdds map[string]registry.MultipartAddedBlob // keyed by string(digest)

[alanshaw]

Again, maybe just an option for no-conclude to existing UploadBlob?

[alanshaw]

Suggested change

	type ParkStore interface {
	type MultipartBlobStore interface {

I mean, you get the idea right?

[alanshaw]

Clever!

[frrist]

Do we need to defer accept?

It's a real architectural fork, and the preliminary sim results say defer. An aborted/expired part is a small, hours-lived blob — the worst churn class in the pdp-sim study (FIL_ONE_FINDINGS.md). Accepted at UploadPart, each one costs AddPieces gas on entry, fil.one-paid proving while it sits in an aggregate, and a remove → compaction rewrite (write-amp double-proving) on exit — costs no storage fee covers. Parked, the same garbage is disk-hours, reclaimed off-chain at zero gas. It's the entry-side mirror of fil-forge/piri#46: a bounded off-chain window at each end of the blob lifecycle (delete SLA there, session TTL/Expires here) so only data that completes into live objects does on-chain work.

By not accepting the Piri operator is storing data and not getting paid.

True. The SP holds parked bytes unpaid until Complete accepts them and pdp pipeline runs — but the exposure is infra-only disk-hours, bounded by the session TTL / allocation expiry below, and per the sim that bounded window is cheaper for every actor than optimistically pushing each part through the pipeline.

There's also an attack vector here ... they have no incentive to not do this ...

Only if billing keys on accepted bytes — and it doesn't have to. Every allocation transits sprue (/blob/add carries space + digest + size; conclude and /blob/abort mark the exits), and sprue is where billing decisions traditionally have lived. Bill open allocations — you pay for reserved space from UploadPart until complete/abort/expiry — and the incentive gap closes at exact AWS parity. Two pieces are missing, both sprue-side and neither a protocol change:

1. materialization — sprue registers blobs only at accept (blob_registry); parked allocations exist only in the agent_index token log, which isn't space-queryable. An allocations table (open → accepted/aborted/expired) fixes that.
2. termination — billing needs a guaranteed exit, and complete/abort are best-effort (that's AWS's forever-billing footgun). The allocation already carries the deadline: ratify Address.Expires as the reservation lifetime and every party enforces it unilaterally — piri GCs at Expires, sprue closes the billing row at Expires, ingot sweeps sessions before it (invariant: session TTL ≤ Expires, so Complete never lands on GC'd parts). Until then, sprue can already bound SP exposure at the /blob/add choke point: per-space open-allocation quotas + delinquency gating. Note accepting at UploadPart would make this attack worse, not better — it upgrades a bounded, GC-able disk-space attack into on-chain gas + proving spend disproportionate to any storage fee the attacker would owe.

This is at odds with the AWS S3 implementation which charges you for storage of
multipart parts until they are aborted or the session expires (which IIRC by
default it does not).

Depends on the billing choice above — with allocation billing it's parity or stricter: your AWS recollection is right (parts bill forever unless an opt-in lifecycle rule aborts them), whereas here every reservation has a default terminal: the ingot sweep (7d default, configurable) with Expires as the protocol backstop.

---

Assumptions I'm operating on, flagged so they can be attacked individually:

1. Billing design is a product decision, not mine. "Bill open allocations" above is a claim that sprue can (the state transits it), not that we will. Whether clients pay for reservations, and from when, needs a product owner — same open question I flagged on the piri thread.
2. The sim is preliminary and unverified Its churn/lifetime means are assumed, so I trust the relative ranking — defer beats accept-early on churny parts — more than any absolute number. And the deferral benefit scales with the multipart abort/expiry rate, which we have no production data for: completed sessions cost the same on-chain either way, just later. If real abort rates are ~0, this is mostly moot and the attack economics are the whole argument.
3. Expires as reservation lifetime is my reading, not ratified. Today it's minted as the PUT-address deadline and probably sized like one (minutes, not days). Treating it as the allocation's billing/GC deadline needs an RFC blessing and value co-design.

If any of these fall, the conclusion moves — genuinely open questions, not rhetorical
ones.

[hannahhoward]

so let me break down this as I see it:

1. From a product perspective, you should pay for multipart uploads parts as soon as you request the space. Alan is right there's an attack vector and I don't want to get burned by that.
2. IMHO what @frrist proposes is somewhat reasonable -- as soon as you reserve space on our network, you should pay for it cause it's space not going to someone else. Which would correspond with blob/add+/blob/allocate.
3. We've traditionally waited for /ucan/conclude + /blob/accept, because we wanted to make sure you actually uploaded before you got charged. I think that made more sense in the storacha case where most of the clients were random JS clients that were on less stable internet connections and who knows what could happen, and more importantly there was no explict way to say "CANCEL!". Now, with the new /blob/abort + /blob/reject flow in implement blob/release and blob/reject; migrate jobqueue into harmony task engine piri#30 plus the fact that most of the non-multipart /blob/add flows are actually a local file not even moving over a WAN, I think doing it on allocate actually maybe makes sense? Even in a non-ingot context with WAN transfers it would make sense in a worse case scenario to call /blob/abort if your upload failed.
4. We still do need to have some kind of cleanup mechanism for multipart -- we could either implement the whole AWS configurable expiration, or we could just set a reasonable default.
5. We still ought to think about the provider getting paid case and whether they should really defer blob/accept. Taking the 50TB upload case, this seems like a situation where finishing the upload could be days not hours, and I wonder if it's in their interest to add the parts tp a dataset... not sure the math on this. Either way, having a defined window on multipart, perhaps based on size, seems important to me -- if we're going to have them allocate 50TB, they can't allocate it for that long without getting paid, so perhaps we just have a default "max days" for multipart and use it to cleanup ingot and piri.

[hannahhoward]

also @frrist I personally don't like the "park" metaphor -- I feel like this is both a CAR thing and Curio thing and left to my own devices I would use a different name. But I'm not going to force this.

[alanshaw]

I just think that if you're going to go through the trouble of creating a multipart request, you're likely going to complete it eventually. Multipart is for data that is bigger than 5GB. I just don't think this is the kind of thing that churns much because there's a big time investment in uploading it. i.e. I'm betting that users don't often decide to stop uploading 3TB of data half way through...

One thing to bear in mind is that a typical multipart upload will be filling aggregates entirely so an abort is not root removals + re-aggregations, it's just root removals.

Finally, (as Hannah pointed out) if you're uploading a huge dataset, I think it's unacceptable (no pun intended) to be using space on a storage node and not paying for it for days on end.

[alanshaw]

I have since learned that most official S3 clients (albeit high level transfer clients) will switch to multipart requests for any data that is over 8MB in size (although varies from 5MB to 16MB). With that in mind it seems likely that we'll see more allocations without accepts (for uploads that are canceled) than I had originally thought and the existing direction and proposal to make the change to charge on allocation seems perfectly reasonable.