GHCR 429s for LinuxServer images: bound registry concurrency and avoid unnecessary tag enumeration

[magrhino]

Problem

WUD can generate a burst of registry requests large enough for GHCR to return HTTP 429 during a scheduled scan. Reports of this behavior commonly affect groups of ghcr.io/linuxserver/* images at the same time, while other checks may still succeed.

The current WUD 8.3.1 scan path appears to amplify registry request volume:

1. The Docker watcher checks containers concurrently with Promise.all(...).
2. findNewVersion() calls registryProvider.getTags(container.image) even for a non-semver tag such as latest, where the useful operation is a digest comparison for the current tag.
3. getTags() follows every registry pagination link.
4. Digest watching then adds manifest requests.
5. Registry requests surface HTTP 429 immediately; there is no bounded retry/backoff that honors Retry-After.

The result can be a large concurrent burst of tag-list pages and manifest calls, leaving throttled containers unchecked until a later successful scan.

Why LinuxServer images are an important example

LinuxServer.io is one of the most widely used Docker image publishers, so this workload is worth accounting for explicitly rather than treating it as an unusual edge case.

Many LinuxServer repositories have deep tag histories. For example, the public GHCR endpoint for linuxserver/bazarr returns a pagination link even when requesting 1,000 tags, meaning WUD must make additional requests to enumerate the complete tag list. Watching several LinuxServer images multiplies that pagination cost.

LinuxServer images expose the problem disproportionately, but the underlying behavior can affect any popular publisher or deployment watching many images from one registry.

Expected behavior

A scheduled scan should avoid overwhelming one registry, and a transient 429 should not leave a group of popular images unchecked without a bounded retry.

Suggested fix / configurable settings

Could WUD implement the following?

• Skip full tag enumeration when the configured tag is non-semver and only the current-tag digest is needed.
• Add a configurable per-registry concurrency limit, with a conservative default.
• Retry HTTP 429 with bounded exponential backoff and jitter, honoring Retry-After when present.
• Preserve successful container results while retrying or deferring only throttled checks.

A targeted per-container check or fair retry ordering could complement this, but reducing the request burst is the primary fix.

Reproduction outline

1. Configure WUD 8.3.1 to watch many GHCR images, including several ghcr.io/linuxserver/*:latest images with digest watching.
2. Run a scheduled watcher scan.
3. Observe registry request volume and HTTP 429 responses affecting some LinuxServer checks.
4. Run again after the throttle window clears; the same checks may succeed without a credential change.

Related issues

This request overlaps with, but is not fully covered by:

• Digest checking should be expanded for "latest" tags and rate limit solution #724 — digest checks and retry prioritization after rate-limit failures
• Watcher doesn't respect cron #611 — watcher runs triggered outside the configured cron and their effect on registry pull limits
• WUD_WATCHER_LOCAL_CRON Config Help/Question #437 — excessive watcher runs and discussion of disabling or scoping Docker-event-triggered checks

Those issues address adjacent sources or consequences of request volume. This issue focuses on the scheduled scan itself: unnecessary full tag enumeration, unbounded per-registry concurrency, and missing 429 backoff.

[xtamtamx]

Confirming this on an independent deployment, plus three data points that may help.

Setup: WUD 8.3.1 (getwud/wud:8.3.1), one Docker watcher, 71 containers, default 0 * * * * cron. Every 429 is from ghcr.io (linuxserver/*, immich-app/*); Docker Hub images in the same run never 429. Each scheduled run produces 8 x 429 - Too Many Requests.

1. Authenticating to GHCR does not help. I configured a GitHub classic PAT with read:packages via WUD_REGISTRY_GHCR_PUBLIC_USERNAME / _TOKEN, confirmed it loaded at startup, and verified WUD's own auth scheme by hand — Authorization: Bearer base64(PAT) returns HTTP 200 on both /v2/<image>/manifests/latest and /v2/<image>/tags/list. Two full post-auth cron windows still produced exactly 8 x 429 each. So "add credentials", which resolves the Docker Hub cases (#248), does not resolve this one.

2. Docker Hub is spared because it is the only registry with a request-reduction override. Registry.shouldWatchDigest() returns true by default, and Hub.ts overrides it to watchdigest: joi.bool().default(false) with an explicit "may result in throttled requests" warning. GHCR inherits no equivalent, so GHCR containers pay the extra manifest calls that Hub containers don't. The precedent for a registry-level knob already exists (PR #1000) — it just was never extended past Hub.

3. Small correction on the pagination point: Registry.getTagsPage() sets const itemsPerPage = 1000 and requests ?n=1000, and ghcr.io does honour it — walking linuxserver/sonarr by hand gives 6 full pages of 1000 plus a final 266, i.e. 7 tag-list requests for that one image, with no Link header on the last page. The amplification is real, just at that scale rather than a smaller page size.

I have not been able to pin down why the scheduled runs are the ones that get throttled. Event-triggered runs of the same watcher over the same 71 containers, minutes later, return 0 errors — but they go through the same watchCronDebounced → watch() path and issue the same requests, so I can't attribute the difference to WUD's request volume alone.

Separately, I've opened #1137 for what happens after a 429: watchContainer() deletes container.result before the call and doesn't restore it on error, so the container is stored and reported as updateAvailable: false rather than "not checked". On my deployment the reported update total drops from 10–11 to 5 on runs with 8 errors. That's downstream of this issue rather than part of it, but it's why the throttling is easy to miss.

---

[xtamtamx]

Follow-up with what looks like the discriminator.

I checked whether the same containers fail every run. Over 5 consecutive hourly runs (71 containers watched, 28 of them on ghcr.io): 6 containers failed all 5 runs, 2 failed 4 of 5, 1 failed once. So it's a near-stable set, not random loss.

But the stable set isn't "the ghcr containers" — only 9 of my 28 ghcr containers ever fail. The discriminator appears to be tag-list size:

image	tags	tags/list pages	fails?
linuxserver/prowlarr	12,000+	12+	yes (5/5)
immich-app/immich-server	12,000+	12+	yes (5/5)
linuxserver/bazarr	9,105	10	yes (5/5)
linuxserver/tautulli	5,214	6	yes (5/5)
paperless-ngx/paperless-ngx	217	1	never
gethomepage/homepage	162	1	never
umami-software/umami	145	1	never
recyclarr/recyclarr	129	1	never
unpackerr/unpackerr	26	1	never

(Counted by walking /v2/<image>/tags/list?n=1000 and following Link, authenticated with the PAT. The two 12,000 figures are floors — I capped the walk at 12 pages.)

The split is categorical, with nothing in between: every failing image needs 6–12+ paginated requests per scan, every never-failing image needs exactly 1. Nine tag-heavy images fanned out concurrently is on the order of 60–100+ requests in about a second, which fits the consistent ~8 429s per run better than the raw container count does.

That seems to support the "avoid unnecessary tag enumeration" half of your proposal specifically: all 7 linuxserver containers here are pinned to :latest, so the tag walk is pure overhead — the tag isn't changing and only the digest matters.

One correction to my earlier comment: I suggested Docker Hub is spared because Hub.ts overrides shouldWatchDigest() to default false. That's accurate but it isn't the dominant factor — 19 of my ghcr images inherit digest watching and never fail either. Tag-list volume looks like the stronger variable, so please weight that part of my earlier comment accordingly.

[magrhino]

@xtamtamx I opened #1143 to address the unnecessary tag enumeration described here, and #1142 to address the error and result-handling behavior from #1137. Both changes resolved the respective problems in my local testing. Since you reproduced both issues and provided useful data, I wanted to update this thread in case you would be willing to try the fixes in your environment.

For easier testing, I also published a public, combined multi-architecture image containing both fixes:

ghcr.io/magrhino/wud:dev

docker pull ghcr.io/magrhino/wud:dev

The image is built from the combined dev branch, supports linux/amd64 and linux/arm64, and is intended for testing only. If you are able to try it, I would appreciate hearing whether it reduces the GHCR 429s and preserves the previous update state when a registry check fails.

[xtamtamx]

Tested ghcr.io/magrhino/wud:dev (digest sha256:135e30600abe9eaf31d6625d79bc9e44aa292914a618123beee0f7d22a7a1c6e) on my deployment for 18 hours, with production 8.3.1 running alongside as a control on an offset schedule. I ran the dev build without registry credentials — valid here since I'd already established that authenticating to GHCR doesn't affect these 429s.

Result: roughly a 91% reduction, not elimination.

	prod 8.3.1	dev build
observed	13 h	18 h
total 429s	109	14
rate	8.4/h	0.78/h
hours with 429s	12 of 13	2 of 18

The distribution matters more than the ratio. Production fails every single hour; the dev build was completely clean for 16 of 18 hours, with all 14 of its failures falling inside one 00:00–01:00 window — the same hour production spiked to 20 against its steady 8. That looks like an externally elevated GHCR window catching even the reduced request volume, rather than a defect in the fix.

Update detection is unaffected. Both instances independently reported 99 containers and 27 updateAvailable, so skipping tag enumeration cost no detections. That was the thing I most wanted to check, since a cheaper scan that misses updates would be a false economy.

Per-image breakdown, which I think is the more useful part:

• Fixed (429s on prod, never on dev): linuxserver/lazylibrarian, linuxserver/bazarr, linuxserver/projectsend, linuxserver/qbittorrent — all pinned :latest, i.e. exactly the non-semver case fix(docker): skip tag listing for non-semver images #1143 skips the tag walk for.
• Still exposed: immich-app/immich-server (v3.1.0) and immich-app/postgres (17-vectorchord1.1.1-...) — both semver, so fix(docker): skip tag listing for non-semver images #1143 doesn't apply and they still enumerate 12,000+ tags.

So #1143 does what it claims, and cleanly. But the residual burst source is tag-heavy semver-pinned images, which suggests the per-registry concurrency limit from this issue is still worth doing alongside it rather than instead of it.

For scale: my store is 62 semver / 37 non-semver across 99 containers, so #1143 addresses roughly a third of the watched set — though it's the noisiest third, since :latest is what the LinuxServer images use.

One caveat on my numbers: I didn't test #1142 rigorously. Post-fix error counts were too low to produce a meaningful before/after on result preservation.

[magrhino]

@xtamtamx thank you for the detailed response I am working on a follow-up PR taking into account your findings and suggestions.

[magrhino]

@xtamtamx Thank you for the detailed comparison. The remaining tag-heavy semver case you identified is exactly what the branch in #1153 targets.

It adds a FIFO concurrency limit per configured registry (default 2) in the shared registry request path, plus bounded HTTP 429 retries that honor Retry-After values up to 60 seconds and release request slots during backoff. #1143 remains the non-semver optimization, while #1153 addresses the residual semver request burst.

We reproduced that remaining behavior during a scheduled scan: 51 watches started within 18 ms, 10 GHCR tag-list requests were issued within roughly 3 ms, and nine returned HTTP 429 within the next second. The branch adds coverage for concurrency bounds, queue ordering, registry isolation, retry exhaustion, and Retry-After handling. Based on that evidence and the completed test suite, we believe #1153 closes the remaining scope of this issue.