Skip to content

Three frictions in the audit→close cycle: drift suppression that doesn't apply, invented audit_notes paths, approve leaving status: draft #378

Description

@montfort

CLI version: 3.39.0

Three frictions hit in a single charter cycle (declare → execute → audit with 3 external auditors → consolidate → remediate → close). Filed together because they share the same context; happy to split if you'd rather track them separately. Ordered by how much time each cost.


1. Drift suppression does not cover omission drift, but the hook says it does

When a charter declares a file that ends up not needing modification, charter drift reports it and the pre-push gate blocks. The gate's own message offers a way out:

  - Document the drift in an AILOG under '## Risk' as 'R<N+1>
    (new, not in Charter)' so the next run suppresses it, or

We did exactly that, and the gate kept blocking. What we had in place:

  • The AILOG documented it under ## Risk as R8 (nuevo, no en Charter), naming the full path (docker/Dockerfile) in the risk text.
  • The charter's frontmatter declared execution_ailogs: [AILOG-YYYY-MM-DD-NNN], so the link existed before the push.
  • The charter also explained it in ## Closing notes.

Still blocked. Reading straymark-pre-pr.sh, the documented behaviour is "AILOG-suppression in the CLI silences alerts on paths already documented as risks in the Charter's originating AILOGs" — which, as far as we could tell, applies to scope-expansion drift (modified, not declared) and not to omission drift (declared, not modified).

The correct resolution turned out to be conceptually cleaner than suppression: if a file was declared and legitimately not modified, it doesn't belong in ## Archivos a modificar — we removed the row and moved the reasoning to ## Closing notes, which is what format v4 provides for. The gate went green and the charter got more honest.

Suggested fix: either make suppression cover omission drift as advertised, or change the hook's message so it doesn't promise a remedy that doesn't apply to the case at hand. The second is cheap and would have saved us three failed pushes. A hint pointing at the "remove the row, explain in Closing notes" resolution would be even better, since that's arguably the right answer for omission drift.


2. charter audit --merge-reports emits audit_notes pointing at paths that don't exist

The emitted YAML fills every audit_notes with a placeholder path that corresponds to nothing on disk:

    - auditor: "glm-5.2"
      ...
      audit_notes: "see audit/charters/CHARTER-26/auditor-primary.md"
    - auditor: "gpt-5-6-terra-high"
      ...
      audit_notes: "see audit/charters/CHARTER-26/auditor-secondary.md"
    - auditor: "qwen3.7-max"
      ...
      audit_notes: "see audit/charters/CHARTER-26/auditor-secondary.md"

The real reports live at .straymark/audits/<CHARTER-ID>/report-<slug>.md — the same path the CLI just read them from. Two additional oddities: the directory prefix is audit/charters/ rather than .straymark/audits/, and two different auditors get the same auditor-secondary.md string, so the field can't even distinguish them.

Since the CLI knows each report's actual filename, pointing at it would be strictly better. We rewrote the field by hand before pasting into telemetry.


3. straymark approve doesn't move status (known, low)

approve writes reviewed_by, reviewed_at and review_outcome: approved, but leaves status: draft:

$ straymark approve AILOG-YYYY-MM-DD-NNN --reviewer "..." --outcome approved
✔ AILOG-YYYY-MM-DD-NNN marked as approved.

$ grep -E '^status:|^review_outcome:' <the AILOG>
status: draft
review_outcome: approved

An AILOG that is simultaneously draft and approved is a contradictory state, and every signed AILOG in our repo sits at accepted, so we edit the field by hand after each approve. Either moving status to accepted on approval, or refusing the combination in validate, would remove the manual step. Filing it mostly so it's recorded — we've been working around it for several charters.


For context on the cycle these came from: 7 execution batches, 3 external auditors from different model families, 0 hallucinations and 0 false positives across their reports. The tooling worked well overall; these are the edges.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions