Add bench generation mode with Harbor and SWE‑bench backends

[Quantumlyy]

Summary

Adds a benchmark generation mode — teich generate --mode bench — that runs reward-labeled benchmark tasks through a pluggable backend (harbor or swe-bench), harvests each native trace routed by its verifier score into output/{passed,failed,borderline}/, and hardens the run for real Docker use (per-task image purge, working Ctrl-C). Everything is opt-in: --mode prompts (the default), github_repo, and all existing behavior are unchanged, bench lives behind the teich[harbor] / teich[swe] extras, and the only new core runtime dependency is jinja2>=3.1 (previously dev-only).

What's included

1. teich generate --mode bench

teich generate grows a --mode {prompts,bench} flag (default prompts) and routes bench through a shared driver + harvest so backends stay thin.

• src/teich/bench/runner.py run_bench(cfg, console, resume, refresh) is imported lazily from cli.py so harbor / swebench stay optional extras; tasks run through a bounded pool sized by max_concurrency (default 1).
• src/teich/bench/backends/base.py defines the BenchBackend protocol (require / tasks / run, where run returns a native trace plus a rewards dict) plus reward routing (primary_score, route_split: score 1 -> passed, 0/unscored -> failed, anything else -> borderline) and harvest().
• Native traces land in output/{passed,failed,borderline}/ with scores + provenance written to output/metadata/<stem>.json; filenames are namespaced bench-<source>-<task>.
• Bench stays its own dataset, never co-mingled with prompts: rows are written as bench-<task>.jsonl, bench/** is added to UPLOAD_IGNORE_PATTERNS, and a mode guard warns against mixing bench and prompt rows.
• src/teich/converter.py excludes bench / verification / sandbox intermediates from the dataset view, while verifier sidecars under metadata/ are still uploaded as dataset provenance — this is what makes the dataset "verifiable": each row carries its task's reward outcome.
• src/teich/config.py adds bench.sources (per-source type / source / split / instances / repo / version / backend) and output.bench_dir (defaults to a sibling bench dir beside traces_dir, outside the dataset); config.example.yaml documents both.
• --resume skips already-harvested bench tasks; --refresh re-downloads remote harbor sources only (swe-bench uses Hugging Face's own dataset cache).

2. Harbor backend

• src/teich/bench/backends/harbor.py runs Harbor-format tasks by driving the harbor package as a library (registry spec, local task dir, or git/HF registry) and reads the verifier's reward dict off the trial env.
• Installed via teich[harbor] (harbor>=0.15.0, Python >= 3.12).

3. SWE-bench backend

SWE-bench ships an evaluation harness, not runnable tasks, so teich runs the agent itself and grades what it produces.

• src/teich/bench/backends/swebench.py checks out the clean repo @ base_commit inside swebench's prebuilt instance image, renders a thin agent layer on top (src/teich/bench/templates/agent.dockerfile.j2), takes the agent's git diff as the candidate patch, and grades it with swebench.run_instance; the resolved verdict becomes the reward (resolved -> passed, unresolved -> failed).
• Installed via teich[swe] (swebench>=4.1.0); teich[bench] = teich[harbor,swe].

4. Ctrl-C actually stops a run

The driver ran tasks in a ThreadPoolExecutor, but SIGINT only reaches the main thread and with ThreadPoolExecutor does shutdown(wait=True), so Ctrl-C looked stopped while Docker ground on in the workers.

• max_concurrency == 1 now runs each task inline on the main thread, so Ctrl-C propagates straight into the Docker call and stops cleanly (the smoke-test / --resume path).
• max_concurrency > 1 cancels queued tasks (cancel_futures) and lets the few already inside a Docker call finish before exit (threads can't be signalled).
• Both paths print a "re-run with --resume" hint.

5. Per-task Docker image purge

harbor's teardown runs compose down --rmi local, which does not remove its custom-tagged hb__<task> image, so per-task images accumulated and filled the disk.

• On every outcome (passed / failed / error) teich removes the per-task image in a finally, gated by output.keep_bench_images (default false = purge).
• harbor: removes hb__<task> read off the trial env, with a deterministic sanitize("hb__" + task) fallback covering the error path where the trial never returned.
• swe-bench: removes the per-task agent-layer image but leaves the shared instance image in place (reused, expensive to re-pull).

6. Templated dataset card

src/teich/trace_readme.py is reworked to render the dataset card from src/teich/templates/dataset_card.md.j2 off a presentation-only context dict — byte-identical to the previous hand-built card, locked by a golden test.

• output.license adds an SPDX id to the frontmatter.
• output.card_extra merges arbitrary extra frontmatter keys (reserved keys dropped, dumped via yaml.safe_dump).
• output.readme_template points at a user .j2 to fully override the card (rendered with the same context); a field_validator errors at config load if the file is missing.
• size_categories is auto-populated from the dataset row count using HF's standard buckets (n<1K ... n>1T).
• jinja2>=3.1 is promoted from a dev-only to a core dependency; both templates ship in the wheel.

Validation

• uv run --extra dev python -m pytest -q -> 612 passed, 27 skipped
• uv run --extra dev python -m pytest -q tests/test_bench.py tests/test_swebench.py tests/test_trace_readme.py tests/test_cli.py tests/test_converter.py -> 168 passed, 2 skipped (bench 31, swe-bench 21, trace_readme 30, cli 29, converter 59)
• Reward routing is covered directly: test_rewards_resolved, test_rewards_unresolved_routes_to_failed, test_rewards_resolved_routes_to_passed
• The card is byte-locked: test_rendered_card_matches_golden_byte_for_byte, plus test_size_category_bucket_boundaries, license, card_extra, and custom-template-override tests
• uv run --extra dev ruff check src/teich/bench src/teich/trace_readme.py src/teich/config.py src/teich/cli.py -> All checks passed
• uv build --wheel -> built teich-0.2.8-py3-none-any.whl; confirmed teich/templates/dataset_card.md.j2 and teich/bench/templates/agent.dockerfile.j2 are packaged

Compatibility

--mode prompts (the default), github_repo, and every existing config key behave exactly as before; bench is opt-in behind teich[harbor] / teich[swe] / teich[bench], the new output knobs all default to today's behavior (keep_bench_images defaults to purge, the card renders byte-identically), and the only new core runtime dependency is jinja2>=3.1 (previously dev-only).

Note: the offline surface (dataset loading, Dockerfile render, reward mapping, backend selection/registration) is unit-tested above; the swe-bench Docker build -> run agent -> grade path is integration-validated only, since CI has no Docker.