Report unredacted AWS account IDs and EC2 instance IDs in PRs

.claude/CLAUDE.md

@@ -190,6 +190,29 @@ Only these extensions are permitted inside **skill** directories (enforced by `s
 - `.kiro/` directories should not be committed.
 - `.DS_Store` and other OS files should not be committed.
 
+## AWS Identifiers
+
+**Redact real AWS identifiers before committing, rather than leaving it to either piece of tooling.** The repository is public and a commit is permanent — removing a value in a later commit does not remove it from the history. Replace an account ID with `123456789012` or `111122223333`, an instance ID with `i-0123456789abcdef0`, and an ARN by rewriting it whole, account and region and resource name together, since blanking the account field leaves the rest of it standing. Do the same for any other real resource name: a cluster, bucket, database, stack or role name describes the environment to a reader and cannot be told apart from an example by automation. See the "Redacting AWS Identifiers" section of [CONTRIBUTING.md](../CONTRIBUTING.md).
+
+Neither piece of tooling is sufficient on its own:
+
+- The skill evaluation tool's redaction pass rewrites the agent's own output and only that. A hand-written `evals.json` prompt is never touched; neither is the `_metadata.json` the harness writes at the root of each functional version directory, whose stack ARNs name the account the run used (gitignored for that reason). A third-party account inside a returned API payload has been observed passing through unredacted while the operating account in the same sentence was replaced.
+- Agent output lands in three files per run: `journal_records.json`, `benchmark.json`, and each scenario's `functional-tests-results.json`, which quotes the output again as the evidence for every assertion. The same identifier usually appears in more than one, so check each rather than fixing the journal and assuming the rest followed.
+- The check reports an account ID only where something proves it is one, so a green check means nothing was proved, not that nothing is there.
+
+A pull request check (`.github/workflows/scan-aws-identifiers.yml`, running `.github/scripts/scan_aws_identifiers.py`) reports unredacted AWS identifiers and applies to every file in the repository, not only to skills. See the "Scanning for AWS Identifiers" section of [CONTRIBUTING.md](../CONTRIBUTING.md).
+
+- An account ID is reported only when something proves it is one; twelve digits on their own are not evidence. Measured across the open pull requests, 54% of the twelve-digit runs on their added lines named no account — a `YYYYMMDDHHMM` datestamp in a resource name, the fractional or integer part of a decimal, a zero-padded counter. The scan therefore runs in two passes.
+- **Pass one gathers evidence and reports nothing**, from exactly three sources: the account field of an [ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html) (the fifth colon-separated field); an object key inside the `text` field of a `tool_summary` block's tool result, in a `journal_records.json` file, where `text` holds JSON as a string and DevOps Agent keys a per-account API result by account ID, so the key itself is the account; and the value of an `aws_account_id` field, in a `journal_records.json` file. The last two are fields of the DevOps Agent journal schema, which is why they are read only in that file name. Other spellings such as `AccountId` and `accountId` are deliberately not read. Pass one reads every changed file whole, so an ARN in an untouched part of a file still proves an account ID an added line names bare.
+- **Pass two reports, on added lines only:** an ARN whose account field holds a non-allowlisted twelve-digit value, reported as the **whole ARN** rather than the account segment (blanking one field would leave the rest of the ARN in place, and a surviving copy of the account ID elsewhere in the file would rebuild it); every occurrence of an account ID pass one proved, in any file type; and an EC2 instance ID as `i-` plus either eight or seventeen hexadecimal characters, which needs no evidence.
+- An ARN with no account field is not reported — `arn:aws:s3:::my-internal-bucket`, `arn:aws:iam::aws:policy/...`. There is no reliable way to tell a real bucket from an example one, since anybody may own `arn:aws:s3:::example-bucket`, so reporting them would be noise; redacting a real one is the author's job and catching it is a human reviewer's. Write example ARNs with `123456789012` or `111122223333`, both allowlisted.
+- Two known gaps. An account ID appearing only as prose in a file that is not a journal is proved by none of the three sources and is not reported. And an ARN is read only where its fields can be lined up, which works for a `*` wildcard and for a CloudFormation `${...}` expression but not for another placeholder syntax in a field before the resource — `{Region}`, `<region>`, `%REGION%` — where the whole ARN fails to parse and a literal account standing beside one goes unreported. Both passes and the lookalikes that must stay silent are pinned by cases in the scanner's `--self-check`, which the check runs on every pull request.
+- **Only the lines a pull request adds are reported on.** `main` already carries real identifiers in committed eval results and example ARNs, so reporting on whole files would fail contributors for content they did not write. Renames are detected, so moving such a file reports nothing.
+- Three ways to resolve a finding: redact the value, removing the whole ARN for an ARN finding; add it to `.github/aws-identifier-allowlist.json` with a mandatory reason; or put an `aws-id-ok: <reason>` comment on the line, which works in the comment-bearing file types only. JSON has no comment syntax, so the allowlist is the route for JSON content.
+- The allowlist parser fails closed: invalid JSON, a missing key, or a blank reason grants no allowance and fails the check, so a typo cannot silently waive a leak. An allowlisted account ID is dropped from the proven set, which silences both bare occurrences of it and any ARN naming it. The skill evaluation tool's redaction placeholders `012345678901` and `i-1234567890abcdef0` are allowlisted, suffixed forms included.
+- The `needs-id-redact` label is applied by automation when the check fails and removed when it passes. Do not add or remove it by hand.
+- Identifiers already on `main` are a separate follow-up change. Do not clean them up opportunistically in an unrelated pull request — the check never reports them, since it only reads added lines.
+
 ## Adding a New Skill
 
 1. Create a new directory under `skills/` with the skill name.

.github/aws-identifier-allowlist.json

@@ -0,0 +1,20 @@
+{
+  "account_ids": {
+    "123456789012": "AWS documentation example account ID",
+    "111122223333": "AWS documentation example account ID",
+    "444455556666": "AWS documentation example account ID",
+    "123456789010": "Example account ID used throughout this repository",
+    "012345678901": "AWS documentation example account ID, and the skill evaluation tool's redaction placeholder for account IDs",
+    "753240598075": "AWS-owned account that publishes the Lambda Web Adapter layer; referenced legitimately in SAM templates",
+    "602401143452": "AWS-owned Amazon EKS ECR registry account for us-east-1"
+  },
+  "instance_ids": {
+    "i-1234567890abcdef0": "AWS documentation example instance ID, and the skill evaluation tool's redaction placeholder for EC2 instance IDs",
+    "i-0123456789abcdef0": "Obvious placeholder, a counting run through the hexadecimal digits; used as an example instance ID throughout this repository",
+    "i-0aaaaaaaaaaaaaaaa": "Obvious placeholder, repeated character",
+    "i-0a1b2c3d4e5f67890": "Obvious placeholder, ascending digit pattern",
+    "i-0a1b2c3d4e5f60011": "Obvious placeholder, the same ascending digits interleaved with letters, ending in a repeated pair",
+    "i-0a1b2c3d4e5f60022": "Obvious placeholder, the same ascending digits interleaved with letters, ending in a repeated pair",
+    "i-0fedcba9876543210": "Obvious placeholder, descending hexadecimal pattern"
+  }
+}