feat: blob removal — serve /blob/remove, /upload/remove, /blob/abort (FIL-522) - #33
Merged
Conversation
github.com/fil-forge/libforge aac837a — Space on RemoveArguments plus the /blob/abort (client to upload service, Cause required) and /blob/reject (upload service to storage node) bindings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bort (FIL-522) /blob/remove releases a space's claim on an accepted blob: sprue deregisters the space's registration and forwards /blob/remove to every provider holding the blob. /upload/remove composes it over an upload's shards. /blob/abort abandons an in-flight upload of a parked (never-accepted) blob: the provider is recovered from the Cause receipt chain (a parked blob has no registration or acceptance to look up by) and the abort is forwarded as /blob/reject under the provider's registration delegation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frrist
changed the title
This was referenced Jul 17, 2026
…pted Align with the blob-removal RFC's per-leg commands: the provider-rooted translation of a client /blob/remove is now /blob/release — the piri client gains Release/ReleaseInvocation and the remove handler forwards it. The client-facing /blob/remove and /blob/abort arguments no longer carry the space; it was already read from the invocation subject. Abort now returns the RFC's named failures instead of generic execution errors: a cause that does not resolve to a known /space/blob/add task fails with MissingCause, and a node refusing the translated reject because the space accepted the blob has its BlobAccepted re-surfaced in the abort receipt, so clients can distinguish 'use /blob/remove' from a retryable fault. Both names come from libforge, shared with piri. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alanshaw
approved these changes
Jul 27, 2026
| // ReleaseRequest contains the parameters for a /blob/release invocation. | ||
| type ReleaseRequest struct { | ||
| Space did.DID | ||
| Digest []byte |
Member
There was a problem hiding this comment.
Shouldn't we use the expected type for this - multihash.Multihash?
Member
Author
There was a problem hiding this comment.
Yep, good call — done in 57c38ef. The pre-existing request structs in this file (AllocateRequest, AcceptRequest, ReplicaAllocateRequest) had the same []byte pattern, so I converted all five to multihash.Multihash rather than just the two new ones (plus the forwardBlobRelease helper param). Assignment-compatible both ways, so no caller changes.
| // RejectRequest contains the parameters for a /blob/reject invocation. | ||
| type RejectRequest struct { | ||
| Space did.DID | ||
| Digest []byte |
Member
There was a problem hiding this comment.
Also here? multihash.Multihash?
Member
Author
There was a problem hiding this comment.
Done in 57c38ef (see thread above — applied to all the request structs in this file for consistency).
frrist
added a commit
to fil-forge/libforge
that referenced
this pull request
Jul 27, 2026
Splits the provider leg out of `/blob/remove` into a new `/blob/release` binding, per the blob-removal RFC ([fil-one/RFC#13](fil-one/RFC#13), merged): `/blob/remove` and `/blob/abort` are now space-subject client verbs whose arguments no longer carry the space (it is the invocation subject), while `/blob/release` carries `{space, digest}` under the provider subject, matching `/blob/allocate` and `/blob/accept`. Cause references say `/blob/add` throughout — the `/space` command prefix died with the UCAN 1.0 transition. Defines the RFC's named errors where every service can share them: `BlobAccepted` (reject refused because **the invoking space** accepted the blob — the guard is space-scoped, not digest-scoped, so another tenant's acceptance never strands a parked allocation) and `MissingCause` (abort's cause is missing or doesn't resolve to a known `/blob/add` task). ## Landing order **This PR is the root of the removal chain** — fil-forge/piri#30, fil-forge/sprue#33, fil-forge/smelt#19, and fil-forge/ingot#40 all pin this branch's head (`3e5e6ba`) and re-pin to the merge commit once this lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Review feedback on #33: the digest is a multihash end-to-end (libforge argument structs and the store interfaces both use multihash.Multihash); the piriclient request structs were the only link in the chain erasing it to []byte. Applied to all five request structs, not just the two new ones, so the file stays consistent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Main's libforge bump (7/24) predates the libforge#49 merge (7/27), so resolve go.mod/go.sum by pinning v0.0.0-20260727220215-5e299c46f62f — libforge#49's merge commit — as the PR description planned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
frrist
added a commit
to fil-forge/piri
that referenced
this pull request
Jul 30, 2026
… task engine (#30) Adds the removal half of the blob lifecycle and moves the aggregation pipeline onto the harmony scheduler. Aligned with the blob-removal RFC ([fil-one/RFC#13](fil-one/RFC#13)). **Protocol**: `/blob/release` — the claim-release inverse of `/blob/allocate`, invoked by the upload service translating a client `/blob/remove` — drops a space's claim on an accepted blob (claim + acceptance + allocation deleted; bytes released once no space holds an allocation or acceptance). `/blob/reject` retires a parked blob — the "don't accept" exit of allocate→accept|reject. Its `BlobAccepted` guard is **scoped to the invoking space** per the RFC: only that space's own acceptance refuses the reject; another tenant's acceptance of the same content-addressed bytes just drops this space's allocation and retains the bytes (multi-tenant liveness). Both named errors come from libforge, shared with sprue. The registration delegation grants `/blob/release` + `/blob/reject` to the upload service. Physical deletion is always deferred to a sweep that re-verifies claims before every destructive step. **Pipeline**: the commp/aggregator/manager jobqueues (and their separate aggregator_db) are replaced by harmonytask tasks — `PDPCommP`, `PDPAggregate`, `PDPAddRoots`, `PDPRemoveSweep` — with all state in harmonydb (`pdp_blob_pipeline`, `pdp_root_submissions`). Stage transitions are transactional with the tables the removal machinery inspects, which closes the accepted-blob removal races by construction: pre-commp rows cancel transactionally, the fold locks its candidates (a row is either cancelled or folded, never both), staged pieces ride the root lifecycle, and roots retire on-chain (`schedulePieceDeletions`) only when every live sub-piece is pending removal. **Validated**: full suite including a harmonydb-backed (Postgres testcontainer) regression harness for the removal races — including the cross-space reject case the RFC mandates; smelt e2e with the injected binary — upload/retrieve plus the full proving loop through the new pipeline, proof verified by the PDPVerifier contract. **Landing order**: fil-forge/libforge#49 (the `/blob/release` binding + shared named errors; pinned at its branch head, re-pin on merge) → **this**. Downstream: fil-forge/sprue#33 (upload-service handlers, forwards `/blob/release`) + fil-forge/smelt#19 (devnet delegations), then fil-forge/ingot#40 (S3 surface). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: ash <alan138@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serves the upload-service half of blob removal, aligned with the blob-removal RFC (fil-one/RFC#13):
/blob/remove(subject = the space, args{digest}) — releases a space's claim on an accepted blob: recovers every provider holding it (primary via the registration's receipt chain, plus non-failed replicas), forwards a provider-rooted/blob/release{space, digest}to each (best-effort), and deregisters last so the receipt chain survives for a retry if every forward fails./blob/abort(subject = the space, args{digest, cause}) — abandons a parked (never-accepted) blob: the provider is recovered from the cause's receipt chain and the abort is forwarded as/blob/reject. A cause that doesn't resolve to a known/blob/addtask fails with the named errorMissingCause; a node refusing because the space accepted the blob has itsBlobAcceptedre-surfaced in the abort receipt, so clients can distinguish "use/blob/remove" from a retryable fault. No local state mutates, so aborts are safely retryable./upload/remove(subject = the space, args{root}) — deletes the upload's root→shards index entry only. Shard blobs are the client's per-digest/blob/removedecision (content addressing shares shards between uploads).All handlers are idempotent (unknown/already-removed → success). Named errors come from libforge, shared with piri.
Landing order
fil-forge/libforge#49 (
/blob/releasebinding + named errors; pinned at its branch head, re-pin on merge) → fil-forge/piri#30 (piri serves/blob/release+/blob/reject) → this — the forwards here invoke piri capabilities that only exist on #30.Same chain: fil-forge/smelt#19 (devnet delegations), fil-forge/ingot#40 (S3 surface). Validated end-to-end by ingot's delete-finality and deferred-multipart itests against piri #30 + this branch.
🤖 Generated with Claude Code