Skip to content

Commit c083bca

Browse files
Merge origin/main into feat/ucan-push-authorization
#330 landed on main, which puts GITLAWB_ENFORCE_OWNER_PUSH=true in the tree this branch builds on — so the delegated-push path here is now the thing that keeps CI keys and agents working, rather than a future improvement. Merged rather than rebased on purpose. The branch was 116 commits behind with 22 of its own, and several of those touch the same three files; a rebase asked me to resolve the same conflicts repeatedly, which is where a wrong resolution gets made quietly. One resolution pass, and the commit history the reviews refer to stays addressable. Five conflicts: - `crates/gitlawb-core/src/lib.rs` — both module declarations kept. - `crates/gitlawb-node/src/api/repos.rs` — main changed the DIDs these handler tests push as; this branch added the `Option<Extension<VerifiedUcan>>` argument. Both. Two further call sites in tests main added after the merge base needed the same argument, which no merge could have supplied: the parameter exists only on this side. Both push as the owner, so `None` is correct. - `crates/gitlawb-node/src/test_support.rs` — git had interleaved two different tests by aligning their shared tail. Rebuilt from main's file plus this branch's 260-line addition (exactly two tests) rather than untangling that by hand; main's tests are byte-identical, verified by hashing the bodies. - `.env.example` and `docs/RUN-A-NODE.md` — main's text says a UCAN `git/push` capability is "verified but not yet honored for authorization", and points at "scoped collaborator / UCAN-delegated push rights" as the work that removes the trade-off. That work is this branch, so keeping main's wording would have shipped a claim this branch falsifies. Main's `true` default is kept; the prose around it now describes delegated push, including the round-eight rule that every link in the chain must name the repository. Verified: fmt, clippy -D warnings, and check --locked --workspace --all-targets are clean. gitlawb-core 116, gl 384, git-remote-gitlawb 64, node auth:: 21 — all green. Both this branch's delegated-push tests and main's `enforced_owner_push_refuses_a_signed_non_owner` pass together, which is the specific thing this merge could have broken. The node suite also reports 118 failures on this Windows host, all in the `ipfs_cid_*` and `sync::*promisor*` classes. They fail in fixture setup with `Os { code: 267, kind: NotADirectory }` before reaching any code under test, and their bodies are byte-identical to main's. Environmental, not from this merge; Linux CI covers them.
2 parents 1ab55e5 + bfc44f9 commit c083bca

49 files changed

Lines changed: 34425 additions & 2770 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 38 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -97,10 +97,15 @@ GITLAWB_REQUIRE_SIGNED_PEER_WRITES=false
9797

9898
# Require the authenticated pusher to be the repo owner on git-receive-pack.
9999
# A valid did:key signature is authentication, not authorization: anyone can
100-
# sign as their own DID. When true, a push is accepted only from the repo owner,
101-
# or from a holder of an owner-rooted git/push UCAN for that repo (see
102-
# docs/RUN-A-NODE.md). Keep false until your pushers are the owner or hold one.
103-
GITLAWB_ENFORCE_OWNER_PUSH=false
100+
# mint a key, derive its DID and sign, so with this off every signed caller may
101+
# push to every repository, private ones included. On by default.
102+
#
103+
# Set to false only for a rolling upgrade whose pushers are not yet the repo
104+
# owner. A holder of an owner-rooted, time-bounded git/push UCAN naming that
105+
# repository can push while this is on — see docs/RUN-A-NODE.md for issuing and
106+
# importing one. A CI key with no such delegation counts as a non-owner and
107+
# cannot push.
108+
GITLAWB_ENFORCE_OWNER_PUSH=true
104109

105110
# Comma-separated libp2p multiaddrs.
106111
# Example: /ip4/1.2.3.4/udp/7546/quic-v1/p2p/12D3KooW...
@@ -195,8 +200,9 @@ GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
195200
# many WRITE-pool slots. This one is load-bearing for push availability, since
196201
# it is acquired before the global write permit: without it, one host minting
197202
# disposable did:key identities could open enough slow pushes to monopolize the
198-
# write pool and 503 every other source (owner enforcement defaults off, and
199-
# the push rate limiter caps arrival rate, not in-flight concurrency).
203+
# write pool and 503 every other source (the push rate limiter caps arrival
204+
# rate, not in-flight concurrency, and owner enforcement can be turned off for
205+
# a rolling upgrade).
200206
# Keyed on the resolved source IP, never the DID, so a DID farm does not defeat
201207
# them; keying granularity follows GITLAWB_TRUSTED_PROXY like the read cap above.
202208

@@ -217,10 +223,26 @@ GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
217223
# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
218224
# Default 4.
219225
GITLAWB_IPFS_WALK_PER_SOURCE=4
226+
# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
227+
# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
228+
# truncated scan sheds a retryable 503, never a false 404. Default 256.
229+
GITLAWB_IPFS_MAX_LEGACY_PROBES=256
230+
# Max repo ROWS one /ipfs request's legacy scan may read from the database. The probe
231+
# ceiling above only counts once a probe runs, and quarantined or private repos are
232+
# denied before that, so without this an all-denying inventory paged the whole repo
233+
# table for one anonymous request. A truncated scan sheds a retryable 503 carrying a
234+
# sealed `continuation` token; echo it as ?scan= to resume, so a holder buried past the
235+
# ceiling is reachable in ceil(repos / ceiling) + 1 requests. Raising this also raises
236+
# the per-caller /ipfs work allowance, since each page is charged to it. Lowering it
237+
# sharpens a coarse oracle: laddering to the end reveals the node's total repo count to
238+
# within one ceiling. Default 2048.
239+
GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS=2048
220240
# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
221241
# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
222242
# skipped without a verdict and the scan continues; if the object is then found
223-
# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
243+
# nowhere the request sheds a retryable 503 instead of a false 404. The effective
244+
# cap is the tighter of this value and the node's internal per-request ceiling of
245+
# 17, so values above 17 have no effect; lower values do tighten it. Default 64.
224246
GITLAWB_IPFS_MAX_REPOS_WALKED=64
225247
# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
226248
# visit costs a repo acquire — on a Tigris cache miss a full archive download,
@@ -241,6 +263,15 @@ GITLAWB_IPFS_MAX_REPO_VISITS=1024
241263
# Must be 1..=3153600000 (100 years): the node derives an Instant deadline from
242264
# this value, and a larger one cannot be represented. Default 600.
243265
GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
266+
# Shorter budget (seconds) for the pre-walk CID resolve: the lookup that maps a
267+
# requested CID to its git oid(s), which runs while the scarce walk admission is
268+
# already held. A well-formed CID with no pin row does no probe and no walk work,
269+
# so without this a stalled lookup could hold a walk slot for the whole request
270+
# budget while nothing walked. The effective deadline is the lesser of this and
271+
# the remaining request budget; walk and probe work stay on the request budget,
272+
# so a slow but progressing scan is never shed by it.
273+
# Must be 1..=3153600000 (100 years). Default 10.
274+
GITLAWB_IPFS_RESOLVE_BUDGET_SECS=10
244275
# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
245276
# from the concurrency caps above). 0 disables. Default 600.
246277
GITLAWB_IPFS_RATE_LIMIT=600

‎Cargo.lock‎

Lines changed: 10 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,9 @@ chrono = { version = "0.4", features = ["serde"] }
5353
uuid = { version = "1", features = ["v4"] }
5454
# http client
5555
reqwest = { version = "0.12", features = ["blocking", "json", "multipart", "rustls-tls"], default-features = false }
56+
# URL parsing (what reqwest::Url re-exports, so the shared redirect predicate can
57+
# take a parsed URL without pulling reqwest into gitlawb-core)
58+
url = "2"
5659
# HMAC
5760
hmac = "0.12"
5861

‎README.md‎

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,53 @@ Notes:
229229
sends the bearer token **only** to your configured origin, never to a URL a
230230
node advertises, so a hostile node can't capture the key or redirect the solve.
231231

232+
### Fetching an object by CID
233+
234+
```bash
235+
gl ipfs list # CIDs this node has pinned
236+
gl ipfs get bafkrei... > object.bin # object bytes on stdout
237+
```
238+
239+
Objects that were pinned before the node started recording which repo they came
240+
from are found by scanning its repo inventory, and that scan stops at the
241+
per-request ceilings in the [Configuration](#configuration) table. A stopped scan
242+
answers 503 with a resume token instead of a false "not found", and `gl ipfs get`
243+
follows the token automatically: up to 8 resumes after the first request, so at
244+
most 9 calls to the node, waiting between attempts for as long as the node's
245+
`Retry-After` asks and never longer than 5 seconds.
246+
247+
The whole ladder runs under a 60 second wall-clock deadline. The deadline bounds
248+
the search, not the download: each attempt gets the time left on it to produce
249+
response headers, and once an object is found its bytes stream outside that
250+
deadline. They are not unbounded, though. The client's own 30 second HTTP timeout
251+
is a total request timeout, running from the start of a request until its body
252+
has finished, so a transfer still going 30 seconds after its request began is cut
253+
off. Waits between attempts never run past the deadline either, so a single run
254+
spends at most around 90 seconds on the network: the deadline plus the 30 second
255+
timeout covering the last attempt. Writing the object out sits outside both
256+
bounds, so piping into a reader that stops reading can hold the command open
257+
longer than that.
258+
259+
Two node-side brakes end a ladder early and are reported rather than retried
260+
around. A 429 is terminal, because the node's rate-limit window is an hour and
261+
that wait cannot be honored inside one invocation; a transient overload (a 503
262+
carrying no incomplete-scan code) is retried on the token already held, under the
263+
same cap, clamp and deadline. The per-IP fanout brake can also stop a ladder well
264+
short of the 9 calls, so automatic resumption is not a guarantee of reaching the
265+
object.
266+
267+
When a bound stops the ladder with a usable token in hand, the command prints the
268+
token and the invocation that continues from it before exiting nonzero:
269+
270+
```txt
271+
resume from where this stopped: gl ipfs get bafkrei... --scan <token>
272+
```
273+
274+
Run that to carry on from where the scan stopped. Re-running without `--scan`
275+
restarts at the first row, reproduces the same truncation and spends the node's
276+
per-IP budget again, so the token is the only thing that makes progress. Tokens
277+
are valid for an hour.
278+
232279
---
233280

234281
## Architecture
@@ -345,7 +392,8 @@ Important node settings:
345392
| `GITLAWB_BOOTSTRAP_PEERS` | Comma-separated HTTP peer URLs. |
346393
| `GITLAWB_P2P_BOOTSTRAP` | Comma-separated libp2p multiaddrs. |
347394
| `GITLAWB_BOOTSTRAP_DISABLE_SEEDS` | Disable embedded seed peers for isolated dev/test networks. |
348-
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
395+
| `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. Defaults to `false` during the staged rollout below. |
396+
| `GITLAWB_ENFORCE_OWNER_PUSH` | Require the authenticated pusher to be the repo owner on `git-receive-pack`. **Defaults to `true`.** A `did:key` signature is authentication, not authorization — anyone can mint a key and sign — so with this off every signed caller may push to every repository, private ones included. Delegated and CI keys count as non-owners: a UCAN `git/push` capability is verified but not yet honored for authorization, so they cannot push while this is on. Set `false` only for a rolling upgrade; see [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md). |
349397
| `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
350398
| `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
351399
| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
@@ -357,16 +405,21 @@ Important node settings:
357405
| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
358406
| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
359407
| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
360-
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Default 64. |
408+
| `GITLAWB_IPFS_MAX_LEGACY_PROBES` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
409+
| `GITLAWB_IPFS_MAX_LEGACY_SCAN_ROWS` | Max repo rows one `/ipfs/{cid}` request's legacy scan may read from the database. The probe ceiling above only starts counting once a probe runs, and quarantined or private repos are denied before that, so this is what bounds a scan over an inventory that denies the caller everywhere. A truncated scan sheds a retryable 503 carrying an opaque `continuation` token; echoing it as `?scan=` resumes the scan where it stopped. Every per-request ceiling on this path (rows, probes, visits, retained rule bytes) mints one, so each request advances the ladder by at least the rows it read and a holder buried past a ceiling is reached in a bounded number of requests, `ceil(repos / ceiling) + 1` when this row ceiling is the one that binds. No ceiling ever produces a 404. Every page is charged to the caller's `/ipfs` work allowance, so raising this raises that allowance too. Lowering it sharpens an oracle: because a truncation emits a token and a completed wrap does not, laddering to the end reveals the node's total repo count (private and quarantined included) to within one ceiling. Default 2048. |
410+
| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks a `/ipfs/{cid}` request may run per phase; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. The effective cap is the tighter of this knob and the node's internal history-walk ceiling of 17 (`MAX_PIN_SOURCES + 1`), so a value above 17 has no effect while a value below it does tighten the cap. It is charged per phase: the provenance lookup and the legacy-scan fallback get separate equal budgets, so one request can run up to twice the cap in total. Default 64. |
361411
| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
362412
| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
413+
| `GITLAWB_IPFS_RESOLVE_BUDGET_SECS` | Shorter budget for the pre-walk CID resolve inside an admitted `/ipfs/{cid}` request: the lookup that maps the requested CID to its git oid(s), which runs while the scarce walk admission is already held. A well-formed CID with no pin row does no probe and no walk work, so without this it could hold a walk slot for the whole request budget while nothing walked, and enough such requests shed every real retrieval at admission. The effective deadline is the lesser of this and the remaining request budget, so a value above `GITLAWB_IPFS_REQUEST_BUDGET_SECS` degrades to the request budget. Only the resolve is on this clock; walk and probe work stay on the request budget, so a slow but progressing scan is never shed by it. Default 10. Accepted range is 1 to 3153600000 (100 years). |
363414
| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
364415
| `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
365416
| `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
366417
| `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
367418

368419
Production note: change the default Postgres password before exposing a node publicly.
369420

421+
Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
422+
370423
---
371424

372425
## Optional node staking

‎crates/git-remote-gitlawb/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ name = "git-remote-gitlawb"
1111
path = "src/main.rs"
1212

1313
[dependencies]
14-
gitlawb-core = { path = "../gitlawb-core" }
14+
gitlawb-core = { path = "../gitlawb-core", features = ["redirect"] }
1515
anyhow = { workspace = true }
1616
reqwest = { workspace = true }
1717
# Reading the node's DID from `GET /` so a delegated push can address its

0 commit comments

Comments
 (0)