cloud bootstrap admin --issue-token always fails: completion audit metadata key issued_token is rejected by the sensitive-metadata filter

[jobiols]

📋 Pre-flight Checks

• I have searched existing issues and this is not a duplicate
• I understand this issue needs status:approved before a PR can be opened

📝 Bug Description

engram cloud bootstrap admin --issue-token can never succeed. The command writes a completion audit event whose metadata includes the key issued_token (a boolean), but the auth-audit sensitive-metadata filter rejects any key containing the substring "token". Since the managed token and its completion audit are persisted atomically, the token creation always fails — making it impossible to issue the first managed admin token via CLI bootstrap.
This also creates a chicken-and-egg lockout: all identity-mutation endpoints (/admin/* and /dashboard/admin/* — create user, issue token, grant project) require a managed admin authenticated with a managed token (requireManagedAdmin checks Source == PrincipalSourceManagedToken; the legacy ENGRAM_CLOUD_ADMIN operator is read-only for identity). The only way to obtain the first managed token is the buggy bootstrap path, and re-running bootstrap is refused once the admin principal exists. Result: a permanently tokenless managed admin with no supported recovery path.

🔄 Steps to Reproduce

1. Run engram cloud serve with ENGRAM_DATABASE_URL and
   ENGRAM_CLOUD_TOKEN_PEPPER configured (authenticated mode)
2. Run:
   engram cloud bootstrap admin --username alice
   --grant-project my-project --issue-token first-token
3. Observe the error above; check cloud_principals/cloud_project_grants:
   admin and grant exist, cloud_principal_tokens is empty
4. Re-run the same command → refused: "a managed admin already exists"

✅ Expected Behavior

engram cloud bootstrap admin --issue-token <name> should complete successfully:
create the managed admin principal, apply the requested project grants, mint the
managed token atomically with its completion audit event, and print the raw token
once. The audit event's own metadata (which only contains a boolean flag
issued_token: true, never the token value) should pass the sensitive-metadata
filter.

❌ Actual Behavior

Token issuance always fails — the bootstrap's own success/completion audit is
rejected by the sensitive-metadata filter because the metadata key issued_token
contains the substring "token":

engram: cloud bootstrap admin: create token with completion audit:
cloudstore: auth audit insert failed:
cloudstore: sensitive auth audit metadata is not allowed: issued_token

Since the token and its audit are persisted atomically, no token is minted. The
admin principal and project grants ARE durably created before the failure, so
re-running bootstrap is refused ("a managed admin already exists"). Because all
identity-mutation endpoints (/admin/, /dashboard/admin/) require a managed
admin authenticated with a managed token — the legacy ENGRAM_CLOUD_ADMIN operator
is read-only for identity — the deployment is left with a permanently tokenless
managed admin and no supported recovery path.

Operating System

Linux (Ubuntu/Debian)

Engram Version

v1.19.0

Agent / Client

Claude Code

📋 Relevant Logs

💡 Additional Context

No response

[andresfv-dev]

We reproduced this on a fresh self-hosted v1.19.0-compatible deployment. Removing issued_token from cloudBootstrapCompletionMetadata allows the initial managed token to be issued. A regression assertion that audit metadata does not contain issued_token passes, along with go test ./.... We have a small local patch ready to contribute once the issue receives status:approved, respecting the repository workflow.

[mtriggiano]

Another data point, plus a patch ready to submit.

We hit this on a self-hosted deployment and have been running a patched image in production since 2026-07-14 (about a month), so I can confirm the root cause and that the fix holds up under real use.

Root cause

issued_token in cloudBootstrapCompletionMetadata (cmd/engram/cloud_bootstrap.go:261) is a plain boolean flag, but cloudstore.sensitiveAuthAuditKey (internal/cloud/cloudstore/identity.go:956) screens audit metadata keys by substring, rejecting anything containing token / authorization / cookie / secret / hash / password / bearer, with token_prefix as the only exact-match exception. So issued_token is a false positive: the completion-audit insert is rejected and the whole --issue-token transaction rolls back, leaving the admin created but without a token.

Worth noting the bug shipped because the CLI test fake doesn't run the real audit guard, so nothing caught the key collision.

Still present after v1.20.0

Confirming what @juandagalo reported in #603: cmd/engram/cloud_bootstrap.go:261 on current main (509e676) still reads "issued_token": issuedToken.

Patch

Branch: https://github.com/mtriggiano/engram/tree/fix/cloud-audit-issued-token-key-collision-main (based on main @ 509e676)

The change is minimal — rename the key to issued_credential (no readers anywhere, git grep confirms a single occurrence), plus a comment documenting the constraint so the next person doesn't reintroduce it, plus a regression test asserting the completion metadata contains no key the sensitive-audit filter would reject.

cmd/engram/cloud_bootstrap.go      | 11 ++++++++---
cmd/engram/cloud_bootstrap_test.go | 24 ++++++++++++++++++++++++

Testing

• go test ./cmd/engram/... ./internal/cloud/... — all pass, including internal/cloud/cloudstore
• go test ./... — the only failures are 10 pre-existing ones in internal/mcp (unknown_project), identical in count and name on unpatched main @ 509e676, so unrelated to this change
• In production: engram cloud bootstrap admin --issue-token exits 0 and issues the token, and the admin API token endpoints work

I'd be glad to open the PR as soon as this gets status:approved — happy to defer to @andresfv-dev if their patch is further along, the important thing is that it lands. Just say the word and I'll send it.

[EmilZapata]

v1.20.0 confirmation: stranded installations still need supported recovery

Confirming the already-reported failure on the published v1.20.0 image, whose release tag resolves to revision ba9e46ced152c37a7cb9e576153c41995873e2fc.

Sanitized minimal reproduction

1. Start a fresh authenticated cloud deployment with managed-token hashing configured.

2. Run exactly one official bootstrap attempt:

   engram cloud bootstrap admin --username <admin> --grant-project <project> --issue-token <device>
   

3. The managed admin and requested project grant persist, but the managed-token count remains zero.

4. The token insert and its audit are transactional. The completion metadata key issued_token is rejected by the sensitive-key sanitizer, so token creation rolls back.

5. First-admin bootstrap can no longer run because an enabled managed admin now exists.

6. Managed token-creation routes require managed-admin authentication, while legacy recovery authentication is intentionally unable to mutate managed identity.

Actual behavior

The deployment is left with:

• one enabled managed admin;
• the requested project grant;
• zero managed tokens;
• no supported route capable of issuing the first managed token.

No direct database mutation, bootstrap duplication, or unpublished binary has been used. No data was migrated. The deployment is stopped and the resulting state is preserved.

Two separate requirements

1. Future bootstrap: PR fix(cloud): allow issued_token key in bootstrap audit metadata #663 addresses the sanitizer false positive so a fresh bootstrap can complete.
2. Existing stranded state: PR fix(cloud): allow issued_token key in bootstrap audit metadata #663 does not provide recovery when the admin and grants already exist but no managed token was persisted.

The expected recovery behavior is a supported, bounded path that:

• recognizes the existing enabled managed admin plus zero-managed-token state;
• issues exactly one managed token and displays its raw value once;
• records the recovery through the normal audit mechanism;
• preserves the existing principal and project grants;
• fails atomically if either token persistence or audit persistence fails;
• refuses recovery when any managed token already exists;
• requires no direct SQL, duplicate bootstrap principal, unpublished code, or weakening of managed authorization.

This should be implemented server-side and remain wire-compatible so official clients require no protocol or payload changes.

Named test scenarios

• fresh bootstrap accepts boolean issued_token audit metadata
• existing enabled admin with zero managed tokens can recover exactly one token
• recovery preserves existing project grants
• recovery token and audit commit atomically
• recovery refuses when any managed token exists
• legacy authentication remains unable to perform general managed-identity mutations
• Postgres integration reproduces and recovers the v1.20.0 partial-bootstrap state

Because #597 already describes the unsupported recovery path, please keep its recovery requirement open even if #663 lands, or move that remaining requirement to an explicitly linked canonical tracker before closing #597.

[NatanahelR]

We reproduced this today (2026-08-26) in a self-hosted Docker deployment using ghcr.io/gentleman-programming/engram:latest.

Environment and observed behavior:

• engram version reports engram dev.
• ENGRAM_CLOUD_TOKEN_PEPPER is configured.
• engram cloud bootstrap admin --username <user> --grant-project <project-name> --issue-token fails with:

cloudstore: auth audit insert failed: cloudstore: sensitive auth audit metadata is not allowed: issued_token

• The managed user is visible in the dashboard afterward, but no managed token is available.
• The pulled image does not expose recover-token in engram cloud bootstrap --help.
• The Cloud service remains healthy and legacy authentication continues to work.

This leaves the deployment unable to complete managed-user onboarding. Could the published Docker image be updated to include the merged recovery path from #786, and the bootstrap audit fix from #663, so stranded administrators can recover and fresh bootstraps can issue their first managed token?