refactor(packaging): single source of truth for lambda shared libraries (uv workspace)

[Austin-s-h]

Description

Lambdas consume two first-party shared libraries — quilt-shared (py-shared) and t4-lambda-shared (lambdas/shared) — but each one pins them independently via opaque archive/<sha>.zip URLs, with nothing keeping those pins coherent. This PR gives the shared libraries a single source of truth so a shared-code change lands consistently across consumers, while keeping per-lambda independence a deliberate, opt-in choice rather than an accident of drift.

The problem this solves

Per-consumer SHA pins drift, in two opposite directions, on master today:

• quilt-shared fragments — it's pinned to several different commits at once across its consumers, so they're not on a common version of the same library. This is functional, not cosmetic: CRC64NVME checksum support (shared: Add CRC64NVME checksum type support #4623) ships only in the consumers that happened to re-pin, and is silently absent from the rest.
• t4-lambda-shared freezes — every consumer is stuck on a single Dec-2024 commit (update lambdas to Python 3.11 #4241) while the library itself has moved on repeatedly (BSQIK/Search: Indexing #4422, enable UP0{15,30,33,34,39} in ruff #4606, Add QuiltSync and Benchling Webhook documentation #4640, shared: migrate to uv and Python 3.13 #4652, build(deps): bump pyarrow from 18.1.0 to 23.0.1 in /lambdas/shared #4951). The shared lib's own uv/Python-3.13 migration (shared: migrate to uv and Python 3.13 #4652) has reached zero consumers.
• There's no single place to edit: one consumer pins inline as a PEP-508 URL while the rest use a [tool.uv.sources] table, and the dependency key is spelled both t4_lambda_shared and t4-lambda-shared.

Propagating one shared change today means hand-editing the SHA in up to ~9 consumers; the #4930/#4931 pair last week is the mechanism in miniature (edit py-shared, then a separate follow-up to re-pin a single consumer).

The approach: an opt-in shared space

• uv workspace: adds a root pyproject.toml + consolidated root uv.lock; members resolve the shared libs from one coherent in-tree source — drift becomes structurally impossible for anything in the shared space, and a shared change is one edit instead of N.
• Independence preserved at the boundary: lambdas that need to move on their own stay standalone with their own lock (via path = "../shared"). Joining the shared space is a choice, not a mandate.
• quilt3 stays out, on purpose: it keeps its own uv.lock and >=3.10 floor and is referenced only as a path dependency — preserving the quilt3-vs-lambdas cadence split.
• Workspace-aware build pipeline: lambda packaging/deploy flows use a uses-workspace helper that reads the members list from the root pyproject.toml at runtime, so the zip and ECR paths branch generically (no per-lambda special-case).
• Python support: quilt3 drops Python 3.9 and continues to support 3.10+ (requires-python = ">=3.10"). Workspace-rooted lambda/shared packaging is on the higher baseline (generally >=3.13, with lambdas/shared at >=3.12).

Open design question

This PR sets the dependency-management model the rest of the series inherits, so it's worth settling deliberately. The core team has reasonably noted that per-component isolation is a deliberate choice for differing change cadences — see the discussion thread for the tradeoff and the alternatives I'm open to (narrowing workspace membership, or keeping per-consumer pins but replacing the opaque SHAs with enforced git + tag pins). Feedback on scope very welcome.

Validation

• uv lock --check across the root workspace and standalone members
• uses-workspace classification verified for every deploy-matrix entry (member vs standalone), both zip and ECR paths
• CI/workflow updates validated for workspace-aware packaging flows

TODO

• Unit tests
• Security: Confirm that this change meets security best practices and does not violate the security model
• Open and Embed: Confirm that this change doesn't break Open variant and Embed widget
• Documentation
  • run optipng on any new PNGs
  • Python: Run build.py for new docstrings
  • JavaScript: basic explanation and screenshot of new features
  • Markdown somewhere in docs/**/*.md that explains the feature to end users (said .md files should be linked from SUMMARY.md so they appear on https://docs.quilt.bio)
  • Markdown docs for developers
• Changelog entry (skip if change is not significant to end users, e.g. docs only)

Greptile Summary

This PR replaces per-consumer SHA-pinned archive URLs for quilt-shared and t4-lambda-shared with a uv workspace at the repo root, eliminating version drift across ~9 lambda consumers. Eight packages join the workspace (py-shared, lambdas/shared, and six lambdas); standalone lambdas that need independence retain their own lock files with path = \"../shared\" local sources instead of opaque ZIP URLs.

• Root workspace + guardrails script: pyproject.toml defines the workspace; .github/scripts/python_packaging.py enforces the dependency contract at CI time and drives workspace-aware uv export / install-target selection in both the zip and ECR build paths.
• Workspace-aware CI and deploy: py-ci.yml and build_zip.sh branch on uses-workspace to choose the correct uv export invocation; the ECR workflow selects build context (repo root vs lambda dir) using the same helper.
• tabular_preview and thumbnail have a build-breaking gap: both are non-workspace-member (so the ECR workflow uses ./lambdas/<name> as the Docker build context), but their pyproject.toml/uv.lock now declare t4-lambda-shared = { path = \"../shared\" } — a directory that is outside the build context. Both uv sync calls in the Dockerfile will fail to resolve this path, breaking ECR deploys for those two lambdas.

Confidence Score: 4/5

Safe to merge for zip-deployed lambdas and the indexer; the tabular_preview and thumbnail ECR builds will fail on the next push to master that touches the lambdas/ tree.

The workspace setup, guardrails script, zip build path, and indexer Dockerfile are all internally consistent. The gap is that tabular_preview and thumbnail switched from URL sources to path = ../shared without joining the workspace, so the ECR workflow keeps building them from their own directory — a context that does not contain lambdas/shared. Both uv sync stages in each Dockerfile will fail to resolve the path dependency.

lambdas/tabular_preview/Dockerfile and lambdas/thumbnail/Dockerfile — both need either promotion to workspace members (so the build context becomes the repo root) or explicit COPY directives for lambdas/shared.

Important Files Changed

Filename	Overview
.github/scripts/python_packaging.py	New packaging helper script: provides guardrails, install-targets, package-name, and uses-workspace subcommands; logic is self-consistent and the guardrail contract enforcement is well-designed
.github/workflows/deploy-lambdas.yaml	Zip deploy correctly mounts the whole repo; ECR deploy correctly selects context per uses-workspace, but thumbnail and tabular_preview are not workspace members while now having path deps outside their build context — Docker builds will fail
lambdas/tabular_preview/Dockerfile	Adds --no-default-groups to both uv sync calls; now depends on t4-lambda-shared = { path = ../shared } which is outside the standalone build context — Docker build will fail
lambdas/thumbnail/Dockerfile	Same issue as tabular_preview — adds --no-default-groups, has t4-lambda-shared = { path = ../shared } path dep outside the standalone build context
lambdas/indexer/Dockerfile	Correctly updated to Python 3.13 base image and restructured to COPY workspace sources (py-shared, lambdas/shared, indexer) before uv sync — workspace-aware pattern works correctly
lambdas/scripts/build_zip.sh	Workspace-aware zip build: branches on uses-workspace for the correct uv export invocation, assembles install_targets correctly for both workspace members and standalone path-dep lambdas; libscipy_openblas strip exclusion is a good addition
pyproject.toml	New root workspace definition with 8 members; uv version pin (>=0.11,<0.12) and workspace sources for quilt-shared / t4-lambda-shared are correctly set
lambdas/shared/pyproject.toml	Lowered requires-python to >=3.12 (enables both workspace and standalone/local-source consumers), relaxed version pins to ranges, cleaned up redundant self-extras in preview optional-deps
.github/workflows/py-ci.yml	Adds guardrails step and workspace-aware test flow; enables the previously-commented-out format check; both workspace-member and standalone export paths are correctly handled

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    ROOT["Root pyproject.toml\n(uv workspace)"]

    subgraph WS["Workspace Members"]
        PS["py-shared (quilt-shared)"]
        LS["lambdas/shared (t4-lambda-shared)"]
        ESI["lambdas/es_ingest"]
        ICE["lambdas/iceberg"]
        IDX["lambdas/indexer (ECR)"]
        MI["lambdas/manifest_indexer"]
        PKP["lambdas/pkgpush + quilt3 path dep"]
        S3H["lambdas/s3hash + quilt3 path dep"]
    end

    subgraph SA["Standalone (path dep)"]
        AC["lambdas/access_counts"]
        PKE["lambdas/pkgevents"]
        PRV["lambdas/preview"]
        TP["lambdas/tabular_preview (ECR)"]
        TH["lambdas/thumbnail (ECR)"]
        TR["lambdas/transcode"]
    end

    ROOT --> WS
    LS --> SA

    subgraph CI["CI / Deploy"]
        ZIP["build_zip.sh"]
        ECR["docker buildx"]
    end

    WS --> ZIP
    WS --> ECR
    SA --> ZIP
    SA --> ECR

    ECR -. "context outside build context" .-> TP
    ECR -. "context outside build context" .-> TH

Loading

_{Reviews (3): Last reviewed commit: "perf: lint" | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 99.70326% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 46.50%. Comparing base (8eec5f4) to head (c868d1c).

Files with missing lines	Patch %	Lines
lambdas/indexer/test/test_index.py	97.36%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4933   +/-   ##
=======================================
  Coverage   46.50%   46.50%           
=======================================
  Files         832      832           
  Lines       34090    34092    +2     
  Branches     5833     5833           
=======================================
+ Hits        15854    15856    +2     
  Misses      16237    16237           
  Partials     1999     1999           

Flag	Coverage Δ
api-python	93.14% <100.00%> (+<0.01%)	⬆️
catalog	21.55% <ø> (ø)
lambda	96.63% <99.11%> (+<0.01%)	⬆️
py-shared	98.02% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Austin-s-h]

@Greptile please re-review

[Austin-s-h]

@copilot Please thoroughly review this PR, including updating the language about Python support. 3.9 has been dropped but 3.10 is still supported.

[Copilot]

@copilot Please thoroughly review this PR, including updating the language about Python support. 3.9 has been dropped but 3.10 is still supported.

Addressed in 1f975a8. I reviewed the PR and updated the changelog language to clarify that only Python 3.9 was dropped and quilt3 still supports Python 3.10+.

[Austin-s-h]

@drernie @sir-sigurd This should be ready to review!

[sir-sigurd]

(AI-assisted analysis; conclusions are my own — sanity-check the specifics.)

The Python-floor handling looks right now (quilt3 staying >=3.10), and the indexer Dockerfile root-COPY is sorted. My one remaining question is the central design choice: the uv workspace trades away independent per-lambda versioning, and I'm not sure that's what we want for independently-deployed lambdas.

Today each lambda pins its own version of quilt-shared/t4-lambda-shared and resolves its own third-party deps in its own lockfile. For services that deploy independently, that independence seems like the property we want: each can bump or hold a shared-lib/dependency version on its own schedule — security fixes, divergent needs, isolated blast radius — and staged migration of the shared libs is possible (one consumer moves, others don't have to).

A uv workspace gives that up. A single shared lock means one version of every dependency across all members (no divergence without evicting a member from the workspace), and all members track the in-tree shared libs rather than a pinned version (so no per-lambda version, no staged migration). That's not a side detail — it's the opposite of how these deploy today.

And I'm not sure there's a problem here that needs it. The current archive/<sha>.zip pins work and are reproducible; their only real cost is co-development friction (you must push the shared lib before a consumer can pin a new version) — but that's the inherent, accepted price of versioned shared code, and it's mitigable (point at a local path while iterating, re-pin the tag to commit).

So the question: what concrete problem does the workspace solve that justifies giving up independent per-lambda versioning? If one-version lockstep across these lambdas is a deliberate goal, that's legitimate — but it should be stated and argued, because it inverts today's model. If it isn't a goal, the current versioned approach already matches what we want; the most I'd change is tidying the drifting archive/<sha>.zip pins into deliberate git/tag pins (stability + legibility), keeping per-lambda version control.

Not blocking — just that this PR sets the dependency-management model the rest of the series inherits, so it's worth settling deliberately rather than defaulting to the workspace.

[nl0]

yeah my thoughts exactly: not sure what are we trying to solve here.
we've revisited most of the packaging pipelines fairly recently, migrating individual pieces to uv,
so the current status quo feels quite workable for the core team.
change cadence is quite different per component, especially for lambdas/py-shared vs quilt3,
so the per-component isolation and explicit cross-component dependencies were deliberate choices directly rooted in our workflows.

[Austin-s-h]

cc @drernie

Appreciate the pushback — it's the right thing to settle here, and it pushed me to quantify what this is actually for. "chore: modernize packaging" undersells it, and I think that framing is why the intent isn't landing. Let me reframe around the problem and back it with numbers from our own history.

The problem: shared-space changes land inconsistently across lambdas

Shared code is pulled in per-consumer via opaque archive/<sha>.zip pins, with nothing keeping those pins coherent. The result on master today:

• quilt-shared (py-shared) is pinned to three different SHAs at once across its six consumers: df53c9ce (es_ingest, indexer, manifest_indexer, py-shared: add utilities for ES ingest #4433), a4b31030 (pkgpush, s3hash, shared: Add CRC64NVME checksum type support #4623), 12bb4a33 (iceberg, py-shared: retarget QueryMaker to per-bucket Iceberg tables #4930) — a spread of ~11 months (Jul 2025 → Jun 2026). These lambdas are not on a common version of the same library.
• That's functional, not cosmetic: CRC64NVME checksum support landed in py-shared at a4b31030 (shared: Add CRC64NVME checksum type support #4623) and therefore ships only in pkgpush and s3hash — the other four consumers silently don't have it.
• t4-lambda-shared (lambdas/shared) is the opposite failure mode — all nine consumers are frozen on one SHA, d496dffb (update lambdas to Python 3.11 #4241, Dec 2024), while the lib itself has moved on at least five times since (BSQIK/Search: Indexing #4422, enable UP0{15,30,33,34,39} in ruff #4606, Add QuiltSync and Benchling Webhook documentation #4640, shared: migrate to uv and Python 3.13 #4652, build(deps): bump pyarrow from 18.1.0 to 23.0.1 in /lambdas/shared #4951). The shared lib's own uv/Python-3.13 migration (shared: migrate to uv and Python 3.13 #4652) ships in zero of its nine consumers.
• There isn't even a single place to edit: thumbnail pins inline as a PEP-508 URL while the other eight use a [tool.uv.sources] table, and the key is spelled both t4_lambda_shared and t4-lambda-shared.

The #4930/#4931 pair from last week is the mechanism in miniature: editing py-shared (#4930) needed a separate same-day follow-up (#4931) to re-pin just the iceberg consumer — and the other five were left on older SHAs, which is exactly why three pins coexist.

What the modernization accomplishes

The workspace isn't "merge everything into lockstep" — it's an opt-in shared space with a consistent git state, and it directly removes the churn above:

• Inherit the shared space, or don't. A lambda co-developing with the shared libs joins the workspace and resolves them from one coherent in-tree source; a lambda that needs to move on its own stays standalone with its own lock (the standalone lambdas already do this via path = "../shared"). Independence stays a deliberate choice at the component boundary.
• Consistent git state by construction. For anything in the shared space, the 3-SHA split and the year-behind freeze become structurally impossible — one source, one resolution, legible at a glance.
• It scales. Propagating a shared change today means up to 9 hand-edited SHA bumps for t4-lambda-shared and 6 for quilt-shared (PR Use master hash for lambdas shared #4251 hand-edited 7 lambdas / 14 files for a single shared bump). In the shared space that collapses to one edit. Adding a lambda is "join or stay standalone," and the build path already adapts automatically — uses-workspace reads the members list from the root pyproject.toml at runtime, so both the zip and ECR paths branch generically (this also closes the hard-coded indexer special-case Greptile flagged).
• quilt3 stays out on purpose. It keeps its own lock and >=3.10 floor and is referenced only as a path dependency — so the lambdas/py-shared-vs-quilt3 cadence split you specifically called out, @nl0, is intentionally preserved.

The tradeoff I'm not hiding

Workspace members do share one third-party resolution, and a few deps differ across them today (e.g. indexer on jsonschema 3.2.0 vs 4.25.x elsewhere, a boto3 range). For deps that should genuinely diverge, the answer is to keep that lambda standalone rather than force it into the shared space — the model supports both. And in fairness, the core team treats per-component independence as a feature, not a bug, and @sir-sigurd reasonably frames the push-then-re-pin step as the accepted price of versioned shared code. So the real question is whether the drift above (three live quilt-shared SHAs, all nine t4-lambda-shared consumers a year-plus behind, the shared lib's own migration never propagated) is worth trading some of that independence for a single source of truth — and I think for the genuinely co-developed cluster it is.

Worth noting the same per-component fan-out shows up beyond shared code: the Dec-2025 uv/Python-3.13 migration shipped as nine separate PRs (#4618, #4647, #4649–#4655), and routine security bumps fan out across the per-lambda lockfiles (idna→3.15 = 14 PRs #4904–#4927, urllib3→2.7.0 = 11 PRs, aiohttp→3.14 = 5 PRs). A single shared lock coalesces a lot of that.

What I'm genuinely open to

I care more about killing the drift than about the workspace specifically, so these are real questions, not rhetorical ones — happy to be steered on any of them:

1. Membership scope. The biggest lever is which lambdas belong in the shared space. If the current set sweeps in lambdas you'd rather keep independent, I'm glad to narrow it to just py-shared + lambdas/shared and the consumers that are genuinely co-developed with them, and leave everyone else standalone. Where would you draw that line?

2. Workspace vs. enforced tag-pins. If the team would rather keep per-consumer pins entirely, I think we can still get the "single source of truth, no silent drift" win without a workspace: replace the opaque archive/<sha>.zip URLs with legible git + tag pins (the pattern we already use for pydoc-markdown in gendocs) and add a guardrail that asserts every consumer points at the same shared tag. That keeps full per-lambda independence and just makes drift a CI failure instead of a surprise. I'm happy to go this route instead if it's a better fit for how you all work — @sir-sigurd, this is close to what you suggested.

3. Sequencing. Regardless of which model we land on, the build-path generalization and the drift cleanup stand on their own. If it's easier to review, I can split those out into a smaller first PR and let the workspace-vs-tag-pin decision land separately, so we're not coupling a mechanical fix to the design call.

4. Python floor. The root currently sits at >=3.13 with lambdas/shared held at >=3.12. If you'd rather align those (or keep more spread for a specific component), I'm flexible — it's a knob, not a constraint.

Does scoping it down one of these ways address the independence concern, or do you read the per-component drift differently than I'm seeing it? Happy to rework in whatever direction gets this to "obviously right" for the core team.

[Austin-s-h]

@greptile re-review

[Austin-s-h]

@sir-sigurd @nl0 @drernie Curious if you have had a chance to discuss this yet? I don't want to base future work off of it unless it is an accepted path. I do think it would largely address #4869 type of behavior on the python side.