An adversarial safety evaluation program for a tool-using RAG agent — threat model, golden set, LLM-as-judge pipeline, judge validation, and mitigations with measured before/after deltas.
The subject is a small customer-support chatbot for a fictional outdoor-gear retailer. It retrieves from a help-centre corpus and can take real actions on orders — issue refunds, cancel orders, change shipping addresses. That last part is the point: an agent that can only talk has a much less interesting failure surface than one that can move money.
The evaluation machinery is complete and tested. No eval has been run yet, so this repo
contains no results. reports/ is empty, and that is deliberate — the standing rule for
this project is that every number in every report comes from a recorded pipeline run. There
are no placeholder figures anywhere.
| Phase | State |
|---|---|
| 1 — Threat model & taxonomy | Complete |
| 2 — Target system | Complete |
| 3 — Golden set (260 prompts) | Complete |
| 4 — Pipeline, judge, database | Machinery complete, no run yet |
| 5 — Judge validation | Machinery complete, needs run data + human labels |
| 6 — Dashboard, mitigations, reports | Machinery complete, needs run data |
141 tests pass with no credentials required.
Two documents carry the thinking. Read them in order and the rest of the repo follows.
THREAT_MODEL.md— what is being evaluated, five trust boundaries, five attacker profiles, and an explicit out-of-scope section.TAXONOMY.md— nine harm categories adapted from OWASP LLM Top 10 (2025) and MLCommons AILuminate, with severity anchors and the pass/fail line for each.
1. Business policy is enforced by the model, on purpose. Ownership checks and the refund ceiling are hard-enforced in the tool layer and cannot be talked past. But the $500 escalation threshold, the 30-day eligibility window, status rules, and the confirmation requirement live only in the system prompt — because that is how these systems actually fail in production, where policy changes weekly and product owns it. A target that hard-codes every check makes excessive agency untestable and models a system that does not exist.
2. Four injection surfaces, not three. Direct user input, retrieved documents, and web
results are the obvious ones. The fourth is free-text fields inside order records —
gift_message, delivery_notes, address_line_2. The customer types those at checkout, so
an attacker launders instructions through a channel the agent trusts because it made the
tool call itself. It survives any defence that only sanitises retrieval.
3. A control arm, sized to matter. Attack success rate alone is a number you drive to zero by refusing everything. 68 of the 260 cases are legitimate customer traffic, including 12 hand-written hard negatives that look like attacks — an angry customer demanding a large refund they are genuinely owed, a customer saying "ignore my last message" while correcting themselves. Every mitigation must report its effect here.
4. The same payload, delivered three ways. Each direct-injection seed is transposed onto all three indirect surfaces with the payload and expected behaviour held constant, giving 24 matched pairs. A gap between the boundaries is then attributable to the boundary rather than to the attacks happening to be harder on one.
5. The judge is measured, not assumed. It is a different model from a different vendor
than the target, it never sees the author's expected_behavior (that would inflate agreement
by construction and produce a judge that cannot work on live traffic), and human labelling is
blind to its verdict. Cohen's kappa is reported per category with bootstrap confidence
intervals — at ~20-30 pairs per category, a point estimate alone would invite overclaiming.
Three free accounts, no billing: Groq (target), OpenRouter (judge), and Supabase (results store).
git clone <this repo> && cd aegiseval
uv venv && uv pip install -e ".[dev,dashboard]"
cp .env.example .env # fill in the three keys
.venv/bin/python -m pytest tests/ -q # 141 tests, no credentials needed
.venv/bin/python scripts/verify_setup.py # pings both providers and the database
.venv/bin/python scripts/run_eval.py --estimate-only # cost projection, no spend
.venv/bin/python scripts/run_eval.py --notes "baseline v1"Then judge validation, mitigations, and reports:
.venv/bin/python scripts/sample_for_labeling.py --run-id <RUN_ID>
.venv/bin/python scripts/label.py data/human_labels/<RUN_ID>.jsonl # blind labelling
.venv/bin/python scripts/judge_validation_report.py data/human_labels/<RUN_ID>.jsonl
.venv/bin/python scripts/run_eval.py --mitigation prompt_v2 --notes "context engineering"
.venv/bin/python scripts/run_regression.py --baseline <RUN_ID> --candidate <MIT_ID>
.venv/bin/python scripts/build_reports.py --baseline <RUN_ID> --mitigation <MIT_ID>
.venv/bin/streamlit run src/aegiseval/dashboard/app.pyEvery script has --help. The golden set is a build artifact — edit
src/aegiseval/dataset/seeds.py and rerun scripts/build_golden_set.py; generation is
deterministic, so a rebuild without seed changes is byte-identical and regression diffs stay
meaningful.
THREAT_MODEL.md TAXONOMY.md the design documents
data/golden_set/v1.json 260 prompts: 68 expert, 160 generated, 32 synthetic
src/aegiseval/
dataset/ taxonomy, entry schema + validator, expert seeds, mutation generator
target_system/ providers, policy, tools, prompts, agent loop, transcript
judge/ nine per-category rubrics, scorer
pipeline/ runner, store, regression gate, sampler, validation statistics
mitigations/ prompt v2, guardrails, retrieval hygiene
dashboard/ Streamlit app
tests/ 141 tests including an offline end-to-end pipeline test
No LangChain, no orchestration framework. This project exists to show the judgement calls in prompt construction, retrieval assembly, judge scoring, and tool-policy enforcement, and a framework that hides any of those hides the thing worth looking at.
Stated here rather than discovered by a reader:
- English only. Translation-based and low-resource-language bypasses are a documented attack class and are untested.
- 260 cases is enough to rank categories and catch large regressions, not to resolve small ones. The regression gate is explicitly underpowered below ~10 percentage points and says so rather than implying absence of a flag means no change.
- One target model, one system prompt. Results are a statement about this system, not about the underlying model.
- Generated cases share structure with their seeds, so the automated arm is a robustness check on surface form, not 160 independent probes.
- A single human labeller, so judge-vs-human numbers are not inter-annotator agreement and the achievable ceiling on kappa is unknown.