Skip to content

Analysis prompt lists a symlinked skill once per harness root (user scope) #202

Closed

Description

@Hoyass

Version: backpass 0.1.30; the same code is still in 0.1.32.
Setup. A shared skill library with per-harness symlinks (the layout the loadSkills comment calls
"the common way to share skills across harnesses"): 154 skills live in ~/.agents/skills, and
~/.claude/skills/<name> and ~/.codex/skills/<name> are symlinks to them. User-scope defaults load
all three roots (USER_SKILLS_DIRS).

Problem. resolveProjectSkillDirs dedupes roots by realpath, but the roots are distinct real
directories, so each shared skill is loaded three times. renderSkillIndexForAnalysis then prints
every copy. In our case the "Project skills (load-on-trigger)" section was 567 lines and ~187k chars
for 199 unique skills - 88% of each analysis prompt, paid on every transcript.

Suggested fix. Keep loadedCopies / description-budget accounting as is (each copy is billed on
purpose), but dedupe only the index shown to the analysis model, e.g. by realpath of the skill file
or by name + description, first path wins:

export function renderSkillIndexForAnalysis(skills) {
  if (!skills.length) return "(this repo has no skills)";
  const seen = new Set();
  const unique = skills.filter((s) => {
    const key = `${s.name}\u0000${s.description || ""}`;
    if (seen.has(key)) return false;
    seen.add(key);
    return true;
  });
  return unique.map((s) => `- ${s.name} (${s.path}) :: ${s.description || "(no description)"}`).join("\n");
}

Measured effect (2 transcripts, backpass analyze --scope user, claude-sonnet-5-5): skill section
187k -> 77k chars, 199 entries each listed once, first-turn tokens per session -43.5% on an equal basis.

Possibly related to #75 (large skill stores inflating prompts), but a different cause: here it is per-skill symlinks across harness roots, not the copied tree.

Activity

  1. kunchenguid commented on Oct 6, 2026

    @kunchenguid
    Owner

    Speaking as Kun's firstmate.

    Thanks for the careful write-up and the measurements. Triaged on latest main 0268201 (chore release 0.1.32 #198).

    Confirmed on main. resolveProjectSkillDirs in src/skills.js dedupes only whole roots by realpath, and user scope loads three distinct real roots (USER_SKILLS_DIRS = .agents/skills, .claude/skills, .codex/skills in src/config.js). loadSkills deliberately follows per-skill symlinks, so one library file reached through three links becomes three entries. renderSkillIndexForAnalysis then prints every entry, and src/analyze.js sends that index with every transcript. loadedCopies and skillDescriptionTokens intentionally bill each copy, and that should stay as it is. Only the index the analysis model reads needs each file once.

    This is different from #75 (the workspace copying the whole skills tree). None of the open PRs (#150, #151, #154, #163, #175, #179, #183, #201) touch the skill index.

    One note for whoever picks this up: please key the dedupe on the skill file's realpath (the same identity loadedCopies already uses), not on name + description. Two genuinely different files that happen to share a name and description are separate skills, and silently hiding one would trade accuracy for size. A test should fail when the index lists a symlinked library skill more than once, and should still list two distinct files that share a name.

    Contract-class: restore. The analysis prompt is meant to show the skill index, meaning the skills that exist. Listing one file once per link is a defect on that existing default path. It adds no new surface, write, or default.

    VISION per-rule:

    • Evidence is the only currency: aligns. The quote and corroboration rules don't change, and coveredBySkill still sees every distinct skill.
    • The human owns the weights: aligns. This is read-only prompt rendering with no change to writes or review.
    • Nothing the model says is taken on faith: aligns. Dedupe by realpath is mechanical, and budget accounting still measures every loaded copy.
    • The budget is the constraint: aligns. Always-loaded billing per copy stays honest, and only the duplicated analysis input goes away (cheaper to verify).
    • It reads what you already have and owns nothing: aligns. Local files only, with no key or server.
    • Failure is loud and named: aligns. The fix belongs at the root (the analysis index), verified against the library-plus-symlinks layout.
    • Scope: aligns. It doesn't widen scope or lower the two-session bar or per-edit review.

    Labeling ready-for-pr.

  2. kunchenguid commented on Oct 20, 2026

    @kunchenguid
    Owner

    Speaking as Kun's firstmate.

    Stale-restamp on latest main c22f2ad (backpass 0.1.37).

    Still real and unhelped on main: renderSkillIndexForAnalysis in src/skills.js (line 255) still prints every loaded entry with no dedupe, so a library skill reached through per-harness symlinks under .agents/skills, .claude/skills, and .codex/skills is still listed once per root in every analysis prompt. No open PR changes the skill index (#212 only imports the function). Not opening a competing PR. The earlier note still applies for whoever picks this up: key the dedupe on the skill file's realpath, keep per-copy budget billing as is, and still list two distinct files that share a name.

    Contract-class: restore (the analysis prompt already promises an index of the skills that exist; listing one file once per link is a defect on that default path, with no new surface, write, or default).

    VISION per-rule:

    • Evidence is the only currency: aligns (no quote or corroboration change; coveredBySkill still sees every distinct skill).
    • The human owns the weights: aligns (read-only prompt rendering).
    • Nothing the model says is taken on faith: aligns (mechanical realpath dedupe; budget still measures every loaded copy).
    • The budget is the constraint: aligns (removes duplicated analysis input while keeping always-loaded billing honest).
    • It reads what you already have and owns nothing: aligns (local files only).
    • Failure is loud and named: aligns (fix at the root of the index).
    • Scope: aligns (does not widen scope or writes).

    Leaving ready-for-pr.

  3. added a commit that references this issue on Oct 25, 2026
    9ded619
  4. added a commit that references this issue on Oct 26, 2026
    b546d7d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-prIssue is in scope and ready for a PR

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions