sqlbench-harness

Benchmark harness for LLM-generated SQL across public benchmarks, with audited setup, execution scoring, and cost reporting.

The harness runs model-generated SQL against local benchmark databases, records provider usage, checks setup inputs for common supply-chain risks, and separates fair runs from diagnostic runs.

What It Does

• Registers benchmark sources for BIRD Mini-Dev, KaggleDBQA, Defog SQL-Eval, and Spider 2.0 DBT.
• Downloads or clones benchmark inputs with provenance records and local manifests.
• Audits ZIP archives for validity and zip-slip paths before extraction.
• Inventories script-like and package-like files in benchmark sources before any execution.
• Wraps benchmark questions, evidence, schema text, comments, and table values as untrusted content in model prompts.
• Runs OpenRouter models, Droid-backed models, or prompt-only dry runs through the same benchmark interface.
• Evaluates SQLite predictions by executing predicted SQL and gold SQL (the benchmark's known-correct query) on the same local database.
• Reports exact-result accuracy, SQL execution errors, token usage, estimated provider cost, and fairness classification.

What It Excludes

• API keys, credentials, and .env files.
• Benchmark database downloads.
• Vendored upstream benchmark repositories.
• Raw provider request and response logs.
• Gold-assisted diagnostic output.

Setup

python3.10 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Set OPENROUTER_API_KEY in .env for OpenRouter runs.

Download Benchmarks

./scripts/bb benchmark setup --name bird-mini-dev
./scripts/bb benchmark setup --name kaggledbqa
./scripts/bb audit

./scripts/bb benchmark list prints the registered benchmarks and expected disk usage.

Run A Model Matrix

./scripts/bb matrix \
  --config configs/models.example.json \
  --benchmarks bird-mini-dev kaggledbqa \
  --track schema-plan \
  --limit 5 \
  --workers 1 \
  --output-stem results/external_sql_matrix_schema_plan

Use --limit for a small bounded run first. For a full-split evaluation, replace --limit 5 with --full; only full-split runs support public ranking claims.

Result Classes

• raw: question, evidence, dialect, and visible schema context.
• schema-plan: raw context plus schema-planning hints derived only from the question text and schema metadata, never from gold SQL.
• diagnostic: any run that uses gold rows, gold-vs-prediction deltas, gold-guided repair, or manual per-case intervention.

Only raw and non-gold tool-assisted runs should be considered for fair ranking. Diagnostic runs are useful for debugging the scaffold and estimating an upper bound.

July 3, 2026 Pilot Evaluation

• Scope: BIRD Mini-Dev and KaggleDBQA.
• Sample: eight models, two datasets, five examples per dataset per model.
• Calls: 80 model completions across the pilot evaluation.
• Failed runs or evals: 0.
• Fairness blockers: 0.
• Estimated provider cost: $0.078203 for 80 model completions.
• Best accuracy: moonshotai/kimi-k2.7-code, 7/10.
• Best low-cost result: deepseek/deepseek-v4-flash, 6/10 for about $0.001143.
• Free baseline: poolside/laguna-xs-2.1:free, 4/10 for $0.

The July 3 result is a pilot evaluation, not a leaderboard claim.

Regression tests and CI

The SQLite regressions workflow runs on every push and pull request with Python 3.13.7 and uv 0.11.19. Actions are pinned to commit SHAs, and the Linux uv archive is verified against its release SHA-256. After runtime setup, checks run offline with no third-party Python packages or benchmark downloads.

Run the same suite from the repository root with the pinned runtime installed:

uv run --python 3.13.7 --no-project --no-sync --offline python -m compileall -q scripts tests
uv run --python 3.13.7 --no-project --no-sync --offline python -m unittest discover -s tests -v

The 12 tests execute the real SQLite evaluator and comparator against temporary fixture databases. They cover denied mutations and unchanged database contents, safe literals/comments/identifiers, instruction limits and connection cleanup, ordered versus unordered results, duplicate counts, types, NULLs, and the versioned evaluation/report contract. A failed test exits nonzero and fails CI. These are local fixture regressions, not measured model accuracy or acceptance of downloaded benchmarks. CI also checks Python syntax and commit whitespace; the repository currently has no configured formatter, linter, or type checker.

Documentation

• docs/HARNESS.md explains the run and evaluation flow.
• docs/SECURITY.md documents setup and dataset safety controls.
• docs/RANKING_POLICY.md defines fair, tool-assisted, and diagnostic result classes.
• docs/RESULTS_2026_07_03.md contains the curated pilot-evaluation table.