Repository navigation
skills(objectstack-data): an eval asks for a hook's refusals catalogue instead of a thrown English sentence - #27088
Merged
Conversation
…atalogue instead of a thrown English sentence
One eval case in hooks-security-seeds-datasources.json: a save refused over
the account's credit limit in an app serving English and Chinese users —
expected `refusals: { CREDIT_LIMIT: { message } }` on the hook, a
`language: 'js'` body throwing `{ code: 'CREDIT_LIMIT' }`, and the zh-CN
sentence under the bundle's `hooks: { HOOK: { refusals } }`; refused: a
sentence on the throw (`new Error('`) and a `handler:`.
Paid down first: five sentences in cases 1-5 that restated their own
`must_not_contain` assertions (453 bytes = 114 units); the case adds
1129 bytes = 283 units; the file is 5616 -> 6292 bytes, 1404 -> 1573
tokens. The ratchet row moves 1404 -> 1573, pinned at the measurement
under the standing ruling quoted beside it. Patch changeset for
`@objectstack/skills`, whose build copies skills/** into its published dist.
Claude-Session: https://claude.ai/code/session_01QSeKYmhU5KEj1nukYhp1Tg
Co-authored-by: Claude <noreply@anthropic.com>
github-actions
Bot
added
size/s
documentation
os-zhuang
approved these changes
Oct 28, 2026
os-zhuang
marked this pull request as ready for review
October 28, 2026 04:58
os-zhuang
enabled auto-merge
October 28, 2026 04:58
Contributor
Author
Contract reviewServed-tier: Read at 2026-10-28T05:04Z by the ① Derived judgments
② Semver level
③ Boundary flags
④ Design judgment
Implemented-by: VERDICT: PASS |
Contributor
Author
维护者速读 — 席位终稿 · PR #27088 · head
|
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #26834
Clause-②: no
Metadata-protocol: none
维护者速读
objectstack-data技能的 eval 文件evals/hooks-security-seeds-datasources.json新增第 7 题:多语言应用里一个 hook 拒绝保存并给用户看一句话,期望答案在 hook 上声明refusals: { CREDIT_LIMIT: { message } }、沙箱jsbody 只抛{ code: 'CREDIT_LIMIT' }、中文句子放进翻译包的hooks组;禁止 body 里抛英文句子(new Error(')和用handler:(没有jsbody 的目录会在保存时被拒)。同文件先删掉 5 句只是复述自身must_not_contain断言的话(453 字节 = 114 token),再按实测把该文件的 token 上限 1404 → 1573(净 +169,席位预算 ≤ +180);@objectstack/skills补一条 patch changeset。refusals目录并在references/data-hooks.md教了它,但没有任何 eval 检查 AI 作者真会用它;照旧习惯直接throw new Error('...')的英文句子会原样到达 zh-CN 用户,而且没有东西在测,技能会悄悄漂移。must_not_contain机械断言覆盖,事实与断言一个不少。skills/objectstack-data/evals/hooks-security-seeds-datasources.json第 7 题(题面、期望、六条 must_contain、两条 must_not_contain)与第 1–5 题的删句;②scripts/check-skills-token-ratchet.mjs该行注释里的算术;③.changeset/26834-hook-refusals-eval.md。node scripts/check-skills-token-ratchet.mjs,预期看到hooks-security-seeds-datasources.json is 1573 tokens (ceiling 1573; headroom 0)与末行82 authored bundle file(s) within their ceilings;node -e "JSON.parse(require('fs').readFileSync('skills/objectstack-data/evals/hooks-security-seeds-datasources.json','utf8'))"无输出即解析通过。skills/**受管面,Tier H;席位不 ready / 不 arm / 不入队)。What changes
skills/objectstack-data/evals/hooks-security-seeds-datasources.json— case 7 joins the six. Prompt: an app serving English and Chinese users; anorderwhoseamountexceeds its account'scredit_limitmust not be saved; API clients branch on the codeCREDIT_LIMIT; a zh-CN rep reads超出账户信用额度. Expected: abeforeInsert/beforeUpdatehook onorderdeclaringrefusals: { CREDIT_LIMIT: { message: 'Over the account credit limit.' } }, a sandboxedbody: { language: 'js', source }that reads the account throughctx.apiand refuses by code only —throw Object.assign(new Error(), { code: 'CREDIT_LIMIT' }), the body's own text never shown — and the zh-CN sentence under the bundle's nestedhooks: { HOOK: { refusals: { CREDIT_LIMIT: { message } } } }, served whole in the caller's locale; a catalogue on a hook without ajsbody is refused at save.must_contain:refusals:,CREDIT_LIMIT: {,code: 'CREDIT_LIMIT',language: 'js',hooks: {,超出账户信用额度.must_not_contain:new Error('(a sentence on the throw — the old habit,throw new Error('Email is required')asreferences/data-hooks.mdstill spells its pre-catalogue examples — and also the half-habitObject.assign(new Error('text'), { code }), whose text is dead),handler:(an in-process handler cannot carry a catalogue;refusalswithout ajsbody is refused at save).Every spelling asserted is the spec's, read on
origin/mainf708cf22:HookSchema.refusalsisz.record(SCREAMING_SNAKE code, HookRefusalSchema { message, status? })(packages/spec/src/data/hook.zod.ts), answered only on alanguage: 'js'body; the body's literalcode: 'X'/e.code = 'X'throws are read at save byhookRefusalCodesThrownIn(hook-refusal-codes.ts); the translation key ishookRefusalMessageKey(hook, code)=hooks.HOOK.refusals.CODE.message(system/i18n-resolver.ts), and the bundle declares it as a nestedhooksrecord (system/translation.zod.ts) — which is why the eval pins the nestedhooks: {and not a dotted path a correct bundle would never contain (the stack's ownhooks:is an array, sohooks: {cannot match it). The teaching inreferences/data-hooks.mdagrees with the spec on every spelling; no disagreement to report.Paid down first (the standing ruling, 「A:先削重复,余额按实测提高」). Five sentences in cases 1–5 that restated their own
must_not_containassertions are gone — the shape the landed objectstack-formula precedent cut: case 1's "Never a dotted path (…400 INVALID_FIELD) and never aformulamirror (…)", case 2's "(an object-levelpermissionskey is rejected)", case 3's "— there is nofield.columnNameon an external object", case 4's three "never" clauses (its one non-asserted clause, "Parents before children in the exported array", stays as its own sentence), case 5's "No validation rule and noreadonlyWhen, …". Each cut spelling remains a mechanicalmust_not_containon the same case; no assertion changed. 453 bytes = 114 units cut; case 7 adds 1129 bytes = 283 units.scripts/check-skills-token-ratchet.mjs— the one row for this file moves 1404 → 1573, pinned AT the measurement (zero headroom), with the arithmetic and the ruling citation beside it as its siblings carry: triage 6450886908 (room inside the package first, no raise) read together with the maintainer's standing ruling 2026-10-27 on the decision card 「skills 修复卡的教学和 eval 都撞 token 天花板,怎么付?」, tap A, verbatim and untranslated: 「A:先削重复,余额按实测提高」. Strictest first: the room was made inside the file; the raise is only the balance trimming could not cover. No other row moved (the open PR #26739 edits other rows;origin/mainwas merged before this PR opened and the row region is disjoint).evals/README.mduntouched (142 / 143, headroom 1 — a mention of refusals does not fit, see Acceptance notes)..changeset/26834-hook-refusals-eval.md—'@objectstack/skills': patch, both declaration lines. Why a changeset and notskip-changeset: the two latestskills/**-only PRs on this eval path (bb60b86 #26416, 7c70f91 #26393) carriedskip-changeset, but the three token-ceiling precedents this card names (b1b153a #26774, 40264be #26810, d7dd47a #26852) each added a@objectstack/skillspatch changeset, and the changeset gate's own text in.github/workflows/pr-automation.ymlsaysskills/**"ships in '@objectstack/skills', whose build copies it into the 'dist' that package's 'files' field publishes … never the label". Measured here:packages/skills/scripts/sync-catalog.mjscopied 93 files intodist/skills/**byte-identical;dist/skills/objectstack-data/evals/hooks-security-seeds-datasources.jsoncarries the newCREDIT_LIMIT(3 hits; positive controlmembershipKeys2 hits; absent-symbol control 0). The eval publishes, so the published-face rule owes the changeset.Two readings for a
skills/**difforigin/mainf708cf2)evals/hooks-security-seeds-datasources.json— bytesceil(bytes / 4))skills/objectstack-data/**— sum of the 17 priced files the gate listsSKILL.mdSKILL.mdtouched)ceil($(wc -c < skills/objectstack-data/evals/hooks-security-seeds-datasources.json) / 4) == ceil(6292 / 4) == 1573.Tests
Head measured:
1a0fb94a.Token ratchet — before, mid (the unpaid reading that proves the gate read the file), after
Published-face probe — `@objectstack/skills` build
Derived local gate families — `dispatch-gates --commands` (60 commands at this head), each exit captured before any pipe, `--ran` reconciled
Edited gate script's own suite (clause ⑤):
scripts/check-skills-token-ratchet.mjshas no*.test.*naming it (git grep -l check-skills-token-ratchet -- '*.test.*'is empty); its suite is its--self-test, run above (79 cases pass). The local-only familiescheck:dev-prereqs/check:skill-frame-freshnessare not owed: nopackages/specbuild and no four-axis frame file touched.test:repois CI's (Test Core); not run locally — the diff touches no vitestrepoproject test.Every family CI runs on the PR is cited from the PR's check-runs by the seat, not re-run here; at report time they are
in_progress.No package build closure (①) is owed: the diff touches no workspace package source (
skills/**at the repo root, onescripts/row, one changeset). No ablation: no tested runtime behaviour changed (eval data and a ratchet constant). No dogfood battery: no!in the title and no(narrowing)arm.Acceptance notes
skills/objectstack-i18nnames nohookstranslation group.content/docs/protocol/kernel/i18n-standard.mdxlistshooks.{hook}.refusals.{code}among the translation targets (PR feat(spec): a hook declares its refusal catalogue, and its body refuses by code, answered in the caller's language #26831 added it), andtranslation.zod.tsdeclares the nestedhooksrecord, butskills/objectstack-i18n/SKILL.md's roster of top-level groups ("apps, messages, globalActions, picklists, dashboards, datasets, reports, pages, flows, positions, permissions, metadataForms, settingsCommon") does not namehooks, and nothing under that skill mentions it (git grep -c hooks origin/main -- skills/objectstack-i18n/is empty). An AI author writing the zh-CN bundle from that skill alone would not know where the refusal sentence goes. A coverage gap in a published skill, no runtime defect; the skills seat decides whether it rides a follow-up (aSKILL.mdrow at its own ceiling, same ratchet shape). Out of this card's file surface.evals/README.mdstill describes this file as "search mirrors via hooks, permission-set scopes, external datasources, seeds (…)"; a mention of the refusal case does not fit its headroom of 1 token and the seat set README at +0. Noted, not changed.references/data-hooks.mdstill spells its three pre-catalogue examples asthrow new Error('Email is required')etc. (lines 163, 780, 785) beside the newrefusalssection — the file is out of this card's surface (the teaching landed with feat(spec): a hook declares its refusal catalogue, and its body refuses by code, answered in the caller's language #26831) and the examples are legal where no catalogue is declared ("A hook that declares norefusalsis untouched"); the eval now scores the habit where a user-visible, translated message is asked for.Verification
node scripts/check-skills-token-ratchet.mjs→hooks-security-seeds-datasources.json is 1573 tokens (ceiling 1573; headroom 0), last line82 authored bundle file(s) within their ceilings.node -e "const f=require('./skills/objectstack-data/evals/hooks-security-seeds-datasources.json'); console.log(f.evals.length, f.evals[6].assertions)"→7and the two assertion arrays above.ceil($(wc -c < skills/objectstack-data/evals/hooks-security-seeds-datasources.json) / 4)→ 1573.Generated by Claude Code