diff --git a/rfcs/2026-07-dynamic-routing-candidates.md b/rfcs/2026-07-dynamic-routing-candidates.md new file mode 100644 index 0000000..84ad715 --- /dev/null +++ b/rfcs/2026-07-dynamic-routing-candidates.md @@ -0,0 +1,133 @@ +# RFC: Dynamic routing candidates + +Status: Experimental + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +[Ingot](2026-06-forge-s3-tenant-management.md#ingot---s3-api) is an S3 facade +that is normally co-located with a Forge Piri storage node. The default Sprue +routing policy selects an eligible storage provider using weighted random +selection among other routing factors. That policy can send an upload accepted +by Ingot to a _remote_ Piri, adding an avoidable data-plane hop and potentially +cross-region transfer. + +Ingot needs a way to express its affinity for its local Piri, that allows for +dynamic updates if needed, and that does not impose significant API changes or +state additions to Sprue. + +## Hypothesis / Goals + +Ingot should be able to dynamically choose a Piri node to write to, providing +opportunity to horizontally scale, manage downtime/maintenance and load balance +writes. + +Non-dynamic space-level affinity (referred herein as _static affinity_), where +ALL writes to a given space are routed to a specific storage node, imposes a +state management burden on Sprue, adds to routing complexity and necessitates an +additional UCAN command for affinity management. It will also disallow +multi-region buckets to exist in the future. + +Adding an optional candidate list to `/blob/add` will allow Ingot to constrain +Sprue's storage-node selection to its co-located Piri(s) without imposing +additional state management on Sprue, all via a minimal backward compatible +protocol change. + +The design has the following goals: + +1. Preserve existing `/blob/add` routing behavior when no candidate list is + given. +2. Allow an invoker to limit a request to one or more registered Piri nodes. +3. Keep the final choice among the allowed nodes with Sprue. +4. Ensure that a candidate-constrained request is never routed to a node + outside its candidate list. + +## Design + +### Terminology + +**Candidates** is a set of storage-node DIDs supplied by an invoker of +`/blob/add`. It is a routing constraint: Sprue selects exactly one node from +the set, so it is not a request that any particular node be selected, unless +the set contains just a single DID. + +**Storage node** is a Piri node registered with Sprue and eligible to receive +blob allocations. + +### `/blob/add` arguments + +`/blob/add` gains an optional `candidates` field: + +```ipldsch +type BlobAddArguments struct { + # Existing /blob/add arguments are omitted. + candidates optional [DID] +} +``` + +An invocation without `candidates` has the same routing semantics as it did +before this change. + +When present, `candidates` MUST contain one or more DIDs. Every DID in the list +MUST identify a storage node registered with Sprue. An invocation with an +empty list or a DID that does not identify a registered storage node is +invalid and Sprue MUST reject it. + +The list is a candidate set, not an ordered preference list. Clients MAY +include one DID to require a particular storage node or multiple DIDs to allow +Sprue to select between them. + +### Routing + +For an invocation with `candidates`, Sprue MUST select a storage node from the +provided list. Sprue MUST NOT route that invocation to a storage node whose DID +is not in `candidates`. + +Sprue MAY use any of its normal routing considerations to choose among the +provided nodes, including availability, capacity, and weights. It MAY choose +any node in the candidate list at its discretion; the order of the list has no +meaning. + +If none of the candidate nodes can serve the request, Sprue MUST fail the +invocation. It MUST NOT fall back to a storage node outside the candidate list. + +A candidate list constrains only the storage node selected for the current +`/blob/add` invocation. It does not guarantee that the selected node remains +available, establish a durable placement policy, or request replication. + +### Ingot use + +An Ingot instance SHOULD include the DID or DIDs of its co-located Piri nodes +as `candidates` on each `/blob/add` invocation it originates. Ingot MAY omit +`candidates` when it intentionally accepts Sprue's ordinary routing policy. + +This preserves Sprue as the authority that selects the node while ensuring that +an Ingot deployment does not cause a blob to be allocated to an unrelated +storage node. + +## Alternatives Considered + +### Static affinity routing + +AKA assigning a "home" node to each space. + +Sprue could store affinity as an attribute of a space as part of Ingot's space +creation procedure. This is additional state that _Sprue_ would have to store +as well as provide additional UCAN command for management (to initially set and +subsequently change). Affinity would also have to be retrieved and considered +when routing, adding an additional complexity to an otherwise relatively simple +decision. + +Conversely, dynamic routing candidates may be useful for use cases outside of +an S3 facade. A client may have a trusted set of providers and wish to manually +route blobs to particular nodes. A space-level policy would make this routing +choice unavailable to them or require additional per-client state in Sprue. + +An additional benefit of dynamic candidates is that Ingot can add, remove, or +replace its local Piri nodes _immediately_ by changing the candidates included +with each write. A static space-level assignment instead requires a separate +update before the change takes effect, and creates a window in which Sprue may +route to an outdated node.