Feat/step76 89 sega pipeline - #1
Merged
Merged
Conversation
… deep-review stack Source for the Step 89 N=100 two-arm experiment (2026-07-03) and every result derived from it (Step 90 semantic-group metrics, Step 92 COLE H2H). Until now this code lived only in the working tree, so no commit could reproduce the paper's main result. Input pipeline (Step 76-81): - crello_preprocessor.py: composite non-text layers onto bg per designer z-order; feed-forward underlay regions (bbox + dominant/contrast colour). - panel_type frame-vs-solid split (opaque coverage < 0.35 = frame) with colour resolution deferred until compositing, so hollow outline panels no longer sample their own white border as a backdrop. - text-as-image: text elements enter as designer-rendered RGBA, which removes the font-fidelity confound and matches the PKU/AesthetiQ "elements given as images" convention. Deep-review stack (Step 82-89): - compose_concept.py / composition_director.py: concept generation with n_concepts override (1 = single-candidate deep review). - layout_metrics.py: neutral geometric measurement anchors (GT-calibrated). - feedback_verifier.py: machine-checkable compliance with colour-distance tolerance instead of exact hex match. - judge_aesthetic.py: issue ledger, LEDGER OVERRIDE, regression re-check, decoupled visual observer behind AGENTLAYOUT_VISUAL_LOOP (default OFF). Measured outcome (see result.md §11, §13): the deep-review stack is net negative against a best-of-3 baseline -- confirmed across two independent judging channels, significant in the COLE channel (sign p=0.032). Committed as the provenance record for that negative result, not as a recommended default. Tests: 457 passed, 12 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two provenance holes, both found while answering "which pipeline version did
Step 89 run on?" -- the honest answer was "unknowable from the artifacts".
1. output2/ was gitignored wholesale, so the drivers that produced every
Step 76-92 number were never version-controlled. Narrow the rule to ignore
results (`output2/*`) while tracking `output2/*.py`. Requires the star form:
git will not descend into an ignored directory to apply a negation.
2. Artifacts recorded neither commit nor model. The only evidence that Step 89
ran on gpt-4o rather than the o4-mini now in config2.yaml lived in prose.
Add provenance.py (git head / dirty / diff sha256 / untracked / pipeline
model / pinned judge model / AGENTLAYOUT_* flags / python version) and wire
it into step89_n100_ab.py and step92_cole_h2h.py.
A dirty run stays identifiable: re-apply diff_sha256's diff to head and you
have the exact source. No timestamp field -- mtime carries it, and a clock
value would make two identical runs differ byte-wise.
step92 reads provenance back from the scoring run instead of re-capturing at
aggregation time; --aggregate-only on scores written before this existed now
reports {"captured": false} rather than misattributing them to today's
environment.
Tests: 457 passed, 12 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…E scores A/B/C axes now come from one protocol on one set of 100 samples (result.md §13, §13.1). - A baseline Smean4 7.120 vs designer GT 7.800 = 91.3%; delta CI [-0.853, -0.512], 11/25/64, sign p<0.001. Significantly behind; not a tie. - B deep-review 6.997 = 89.8%, worse than A on every axis. B-A paired: A better 45 / tie 27 / B better 26, sign p=0.032. - Convergent validity: blind pairwise (12/7/81) and COLE absolute scoring (11/25/64) agree on the same 100 samples through independent channels. - SDL: zero wins in both arms. Judge saturates designer GT at 8/10 (79/100), so the agent can tie but never exceed. Verified against raw score distributions, not a bug. Honest boundaries recorded: 91.3% is not comparable with Step 70's 86.6% (renderer + input protocol both changed); and "B significantly lowers Smean" overstates it -- the sign test is significant while the mean-delta bootstrap CI [-0.316, +0.026] crosses zero. Correct claim: B loses to A more often, by a small margin. Also records that the Step 89 source was reconstructed from mtimes, since that run predates provenance capture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…und curve Adds §14 (Steps 93-95, written in a parallel session) and §14.4 (Step 96, this one). All four are zero-API: they only formalize numbers that already existed. Step 96 (E2') recomputes the per-round paired gain curve on the LEGACY pipeline (Step 73-74 full trace, raw-asset input + refinement loop, R2 renderer) because paper §5.8 describes that pipeline, while the Step 93 curve was measured on step89's text-as-image protocol. The two curves disagree on the first round: transition legacy (n=161) step89 arm A R0->R1 -0.075 p=0.694 +0.440 p=0.054 R1->R2 +0.169 p=0.734 +0.138 p=0.88 "Value concentrates in round 1" is therefore a step89-specific phenomenon, not a property of the system. §11.4's design rule (best-of-3 + one repair round + stop) must be re-scoped to step89 or withdrawn when §5.8 switches sources. All four legacy transitions are non-significant with CIs spanning zero, and mean total is flat across five rounds (33.55 -> 33.32) -- which agrees with, and strengthens, the refinement-loop net-negative result from Steps 20b/31/32/74. Read-path cross-validated: this script's n_pairs (160/142/133/126) match the rounds_observed column of full_result/_aggregate/per_round_convergence.md, produced months earlier by an unrelated script. Sample accounting made explicit: 179 dirs, 1 with no trace, 17 whose trace is an empty list (pipeline never reached a judge verdict) => 161 usable. A paper citing "N=178" must account for the gap. Numbering skips to 96: step95 was taken by the parallel session's protocol map. step93_perround_curve.py and step94_signtest.py are that session's work, committed here so §14.1/§14.2 do not cite untracked code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 91 (o4-mini vs gpt-4o cost experiment) needs the model id known to _calc_usage, which otherwise throws on an unregistered name. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…two paper runs Freezes the provenance of the two runs every main-table number descends from, and narrows .gitignore so the freeze records are tracked while bulky results stay out. Both files are marked capture.method = "reconstructed": neither run recorded a commit or a model, so every field carries the evidence it was inferred from rather than asserting a fact the artifacts cannot support. step89_n100 (2026-07-03, source of A/C axes + Steps 90/92/93): - pipeline code == commit 63dc93d. Evidenced by: no commit existed between 2026-06-25 and 2026-07-08, 63dc93d captured that tree unchanged, and no .py under metagpt/ext/agentlayout has an mtime after the run start. - Sole exception semantic_group_metrics.py (2026-07-08) is off the execution path -- verified by grep that nothing imports it. - Driver drift disclosed: step89_n100_ab.py gained provenance wiring in 9884f3c after the run; arms, judging and scoring untouched. - Model gpt-4o rests on config2.yaml history (prose, not machine-checkable). Stated as such, with confidence noted. step92_cole_h2h (2026-07-09, source of the B axis): - Judge pinned to gpt-4o in code (step21_phaseb_eval.MODEL), so unlike step89 this is machine-verifiable and independent of config2.yaml, which read o4-mini that day. - Records the deliberate jpeg-under-png-header deviation, the fixed bootstrap seed, and the arm-level resume rule. Cross-checked against the artifacts before committing: blind-pairwise tallies, cost, judge model, both pct_of_gt figures, the B-A sign p, and the sample-id sha256 all match what is on disk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… ids Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tical reanalysis (A3-12S)
Zero-cost, read-only over frozen a3-rel100-{t0,t2,t3}-01 artifacts and the
merged human oracle trees; no layout regeneration, no API calls.
1. Tree accuracy (a3.tree-accuracy.v1): T2 predicted trees vs human oracle,
99/100 evaluated + 1 explicit planner failure; same-group F1 0.6044
(primary), parent-child F1 0.3394, semantic-type acc 0.7437; exact
semantic-role acc 0.0 kept as lower bound only. Macro + pooled micro +
sample-level bootstrap 95% CI (seed 20260712, 10k).
2. Statistical reanalysis (a3.relation-stats.v1): per-sample SGC/TLC/PCA
deterministically recomputed (arm means reproduce experiment-log section
23.3 exactly); 9 paired sign tests with Holm + Bonferroni; 8/9 stay
significant after Holm, only T3 vs T2 PCA does not (p=0.0639); 3/300
generation failures listed explicitly; results.md/results.tex included.
Both bundles are write-once with input/code/sample-ID hashes in their
manifests. Tests: 24 new, agentlayout suite 753 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
Replace 49 verbose execution-checkpoint transcripts (2,737 lines) with the current facts only (168 lines): immutable N=100 artifacts + hashes, consumed authorizations, Full-Crello frozen inputs, the batch-001 paused ledger state with exact resume conditions, operational constraints, and stop conditions. No factual state was changed; superseded historical paid commands were removed entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…ff and batch plan User directive 2026-07-12: sessions must read next_step.md and act directly. Removed the standing mandates to re-verify artifact hashes, re-run readiness checks/session-catchup, and run disk/Git/network/process gates at session start; the only pre-flight left is disk >= 80 GiB and no concurrent same-batch process immediately before a paid launch. Batch acceptance is reduced to run summary + published six-axis aggregate + cost within guardrails (write-once manifests already record hashes). Planning ledgers update only at batch/phase milestones. Stop conditions and budget caps are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…l (A3-13P) Zero-cost research-design deliverable per CODEX_HANDOFF §5/§6 row 1: AI_REFERENCE_PROTOCOL.md defines 3 independent MLLM annotators, deterministic consensus + blind 4th-model escalation, ai_reference.v1 provenance/namespace, Alternative Annotator Test, downstream stability gate, MLLM-panel preference protocol, file list, offline tests, and staged call/token/USD budget. No code, no API calls, no artifacts touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01125QEt2L6s4iExxbKJ2uqn
…es are AI-annotated (A3-14C) User disclosed 2026-07-12: all 100 reference tree annotations and both adjudication passes were produced by AI models (hui=GPT-5.6 sol, neiji=Claude Fable 5, nina=Gemini, T=GPT; adjudication=GPT) operated in isolated chat sessions, not by human annotators. source="human_oracle" is a misnomer (= ai_reference.multi-model.v0). Frozen artifacts unchanged; authoritative sidecar ANNOTATION_PROVENANCE_CORRECTION.md + log §27 define reinterpretation rules and required limitations. AI_REFERENCE_PROTOCOL.md marked SUPERSEDED (its human-calibration premise no longer exists). All experimental numbers unchanged; no reruns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01125QEt2L6s4iExxbKJ2uqn
…y GPT-5.6 sol single-model (A3-15B) Read-only, $0 analysis over frozen n80 annotations: adjudicated values match annotator hui (GPT-5.6 sol) on 99.0/99.7/99.8% of contested semantic_type/parent_id/same-group units; 75/80 adjudicated files are structurally identical to annotation_hui.json. Combined with the pilot (oracle = GPT-5.6 sol full re-annotation), all 100 reference trees are effectively single-model GPT-5.6 sol; the multi-model consensus reading in log §27 is retracted. Correction sidecar §6, log §28, provenance mapping finalized (Gemini=3.5 Flash, T/adjudication=GPT-5.6 sol). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01125QEt2L6s4iExxbKJ2uqn
…=100) Adapter suite under layout_agent/external_baselines/elem2design/: prepare_inputs builds the official five-layer conversation JSON from frozen A3 P-Full/R3 inputs (R3 bitmaps + text + canvas + official predicted roles only; recursive forbidden-key scan and per-file R3 hash checks fail closed), infer_patched runs the official checkpoint with documented Pascal patches (4-bit NF4 via quantization_config kwarg, quant-safe missing-weight init, mm_projector/vision_tower/lm_head kept fp16, optional fp32 bnb compute dtype = 4.4x faster on sm_61, KV cache enabled, per-(sample,turn) seeding, cross-shard resume with fsync'd JSONL), convert_outputs maps element index back to asset_id fail-closed, and evaluate/compare reuse the repo's shared human-tree metrics, sega geometry metrics, sign test, Holm, and bootstrap (seed 20260712). Gate 1 load test and 13 protocol unit tests included. Verified: Gate 1 PASS (6.6GB VRAM, LoRA merged into 4-bit), N=1 full-chain probe PASS (18/18 coverage, 0 turn errors), N=5 smoke PASS (5/5 evaluable). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…=100 (A3-13E) E2D completed 94/100 (6 explicit conversion failures, 5 tied to its max_num=25 training bound); A3-T2 98/100; paired N=93. A3-T2 wins all three semantic axes after Holm (SGC +0.168 p=1.5e-08, TLC +0.158 p=7.5e-14, PCA +0.117 p=0.0034) and Overlay (-0.135 p=4.3e-16); Alignment marginally favors E2D (+0.0010, 77/93 ties, p=0.021). Zero paid API; 4x1080Ti, ~70 min wall with fp32 bnb compute dtype. Write-once bundles under layout_agent/evaluations/a3-external/; run artifacts stay untracked. Log section 26. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…(A3-13E addendum) A3-T2 without candidate selection (first candidate / mean-of-3) still beats Elem2Design on all three semantic axes after Holm (SGC p<=1.9e-10, TLC p<=3.8e-09, PCA p<=8.8e-03) with effect sizes equal to or larger than the judge-selected B0 comparison; Ove unchanged; Ali ns (first) / +0.001 (mean3). Log section 26.1; write-once bundle under evaluations/a3-external/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
Implements the two resume mechanisms checkpoint 48 specified but never built: completed-sample skip + --skip-sample operator flag in run_a3.py, and --resume-ledger cumulative replay in A3PaidBudget (authorization-hash verified, unsettled reservations refused, caps never reset; 3 new tests, 7/7 green). Batch 001 finished 98 completed + 2 authorized OperatorSkip; cumulative ledger 720 calls / 2.56M in / 472k out / ~USD 4.05, inside the original USD 7 envelope. Six-axis sidecar published (Ali 0.00096, Ove 0.0911, Rea 0.00225, Occ 0.00689; Und N/A). Log section 27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…13E 26.2) Same per-sample backgrounds + frozen BASNet+ISNet stack; A3-T2 values taken verbatim from the frozen formal SEGA bundle; E2D boxes only. Occ favors A3-T2 (0.00563 vs 0.00734, 16W/75L/2T, Holm(4) p=5.3e-10); Rea is uninformative (both arms 0.0, 93/93 ties, flat backgrounds). Geometry family Holm recomputed over the complete 4-test family, superseding the interim 2-test Holm. All six axes now accounted for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3Zce66bwHcZ5TSDDqDz29
…TS.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015W9ss5LMDbF3aMDNzrukCE
…olation types + A3 active subset) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015W9ss5LMDbF3aMDNzrukCE
…e 5f885a9a, zero-cost from batch-001 artifacts) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X13eo33LLZSTcghCDVMsGK
…orce-add past output2 ignore, same as prior result docs) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X13eo33LLZSTcghCDVMsGK
…c background + multi-member groups + parent-child chain (sample 58ac638c) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X13eo33LLZSTcghCDVMsGK
Replace upstream MetaGPT marketing content (news, Discord, MGX, tutorials) with AgentLayout thesis project overview: A3 pipeline architecture, install/ config, usage entry points, datasets & metrics, repo structure, doc index, related work, and MetaGPT acknowledgment + citation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jiHLfiGT34uoffveyEzJf
Delete .planning/ scratch files and four superseded docs (new_plam, new_experiment, live_runs_table, CODEX_HANDOFF) whose content is consolidated in A3_EXPERIMENT_LOG.md / EXPERIMENT_MATRIX.md / IMPLEMENTATION_LOG.md. Untracked bulk artifacts (full_result per-sample traces, old demo outputs, pku_run, sega_pre, step archives, render logs) were removed from disk in the same cleanup (~9GB reclaimed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jiHLfiGT34uoffveyEzJf
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013jiHLfiGT34uoffveyEzJf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
Feature Docs
Influence
Result
Other