Skip to content

feat(bootstrap): recursive batches, README-validated order, per-file run-as user - #618

Merged
mandy-chessell merged 3 commits into
odpi:mainfrom
dwolfson:feat/bootstrap-recursive-batches
Oct 9, 2026
Merged

feat(bootstrap): recursive batches, README-validated order, per-file run-as user#618
mandy-chessell merged 3 commits into
odpi:mainfrom
dwolfson:feat/bootstrap-recursive-batches

Conversation

@dwolfson

@dwolfson dwolfson commented Oct 8, 2026

Copy link
Copy Markdown
Member

Why

Data Initialization only looked at .md files directly inside each dr-egeria-inbox folder. Nested folders never loaded unless they had their own inbox symlink: 1. coco-data-hub/strategic-digital-products (14 files) never ran. The order inside and across folders had also drifted from the coco-workbooks READMEs, and every file loaded as erinoverview, although the READMEs load each file as the persona who owns it.

What changes

Discovery (bootstrap_batches.py, quickstart + freshstart)

  • Subfolders run inline at their place in the parent's order. A _batch.json files entry can name a subfolder ("sub/") or a file in one ("sub/x.md").
  • New exclude field. README.md and output/plumbing folders (dr-egeria-outbox, logs, data, …) are skipped. The outbox folders hold processed copies with live commands.
  • New userid field: set per folder, or per entry as {"file": …, "userid": …}; the nearest declaration wins. Passwords never go in manifests: they come from EGERIA_BOOTSTRAP_PASSWORD_<USERID>, then EGERIA_USER_PASSWORD, then secret.

No double runs

  • A subfolder that is another batch's root (inbox symlink) runs only in that batch.
  • Run All and auto-heal run each resolved file at most once per pass. A repeat is reported as duplicate, not as a failure.
  • An advisory run ledger (~/.pyegeria/bootstrap_ledger.json) lets the panel show ✓ / changed since last run / no Portal run recorded. It never causes a file to be skipped.

Validation, shown in the admin panel: files whose position was defaulted, stale manifest entries, files with no Dr.Egeria commands (checked against pyegeria's own command list), and overlap between batches. Each file row also shows its run-as user.

Manifests, taken from the READMEs

  • Data Hub: follows the README's step order, including subfolders; prose files are excluded.
  • Per-file users: files load as the persona the README names, across Governance Program, Data Hub, Data Field Naming, Strategic Digital Products, Systems Inventory, Keeping Safe and Data Privacy.
  • Local Dashboards: the three previously unlisted files are now listed.
  • _folder_order.json: Governance Program → Data Privacy → Data Field Naming → Data Hub → Keeping Safe → Sustainability → Martyn's Law → Sales Forecast.

Docs: portal-docs/tools/data-initialization.md and data-initialization-manifests.md.

Testing

  • Live on quickstart: /api/bootstrap/batches returns the expected order, userids and notes for every batch. All canaries stayed present and no heal ran.
  • All ten persona users (juleskeeper, faithbroker, stewfaster, ivorpadlock, tessatube, reggiemint, erinoverview, garygeeke, peterprofile, pollytasker) authenticate on quickstart.
  • Offline, with a fake dr_egeria on PATH:
    • outbox and README files are skipped
    • a symlinked child runs only in its own batch
    • a second batch reaching the same file gets duplicate and the run still succeeds
    • the ledger updates
    • userid precedence and the exact --userid/--user_pass arguments are correct
  • Not run: a real Run All against Egeria with the new users.

🤖 Generated with Claude Code

@dwolfson
feat(bootstrap): recursive batches, README-validated order, per-file …
9e68f47 
…run-as user

Data Initialization only looked at .md files directly inside each
dr-egeria-inbox folder, so nested folders never loaded unless they had
their own inbox symlink (1. coco-data-hub/strategic-digital-products, 14
files, never ran). Order inside and across folders had also drifted from
the coco-workbooks READMEs.

Discovery (bootstrap_batches.py, both quickstart and freshstart):
- Subfolders run inline at their place in the parent's order. A
  _batch.json `files` entry can name a subfolder ("sub/") or a file in
  one ("sub/x.md"); unlisted items are still appended alphabetically,
  but flagged as "position defaulted".
- New `exclude` field. README.md and plumbing/output folders
  (dr-egeria-outbox, logs, data, ...) are never descended into; the
  outbox folders hold processed copies with live commands.
- New `userid` field (folder-level, or per entry as
  {"file", "userid"}); nearest declaration wins. Passwords never go in
  manifests: EGERIA_BOOTSTRAP_PASSWORD_<USERID>, then
  EGERIA_USER_PASSWORD, then "secret".

Double runs:
- A subfolder that is another batch's root (inbox symlink) runs only in
  that batch; its parent skips it.
- Run All and auto-heal run each resolved file at most once per pass
  (a repeat is reported as "duplicate", not a failure).
- Advisory run ledger (~/.pyegeria/bootstrap_ledger.json) records each
  successful run with a content hash, so the admin panel can show
  current / changed-since-last-run / no Portal run recorded. It never
  causes a file to be skipped.

Validation: per-batch notes in the admin panel for defaulted positions,
stale manifest entries, files with no Dr.Egeria commands (matched
against pyegeria's own command list), and cross-batch overlap.

Manifests, from the READMEs:
- Data Hub: README step order including subfolders; prose files excluded.
- Data Governance Program, Data Hub, Data Field Naming, Strategic
  Digital Products, Systems Inventory, Keeping Safe, Data Privacy: each
  file loads as the persona its README names (all ten verified to
  authenticate on quickstart).
- Sales Forecast: prose files excluded.
- Local Dashboards: the three previously unlisted files are listed.
- _folder_order.json (quickstart + freshstart): Governance Program,
  Data Privacy, Data Field Naming, Data Hub, Keeping Safe,
  Sustainability, Martyn's Law, Sales Forecast; alphabetical had
  Sustainability before Keeping Safe and Data Privacy last.

Docs: data-initialization.md and data-initialization-manifests.md.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
@dwolfson
docs(bootstrap): user guide for automatically loading Dr.Egeria files
028c52d 
Adds DR_EGERIA_AUTO_LOADING_GUIDE.md at the repo root: a guide for people
who maintain folders of Dr.Egeria files, covering what changed in Data
Initialization, how files are found and ordered, exclude, run-as users,
when files run (auto-heal, Run Now, Run All), the admin panel's marks
and notes, recipes, and troubleshooting. Linked from
portal-docs/tools/data-initialization.md, along with pyegeria's
dr_egeria_folder, which reads _batch.json with the same rules.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
@dwolfson dwolfson mentioned this pull request Oct 8, 2026
Merged
@dwolfson dwolfson self-assigned this Oct 8, 2026
@dwolfson dwolfson mentioned this pull request Oct 8, 2026
Merged
@dwolfson
fix(bootstrap): point canaries at each batch's last file
2f0d1b4 
Auto-heal re-runs a batch only while its canary is missing. A canary
created by an early file lets an interrupted heal look complete, and
the rest of the batch is never retried. On 2026-10-03 this left
Coco - 1. Data Field Naming without 6 of its 33 files: its canary came
from file 1 of 33, and an app reload killed the heal after common.md.

Each canary now names an element created by the batch's last file, or
the last file that creates anything. Every new canary was confirmed
present in the live Egeria first, so the switch can't trigger a heal:

- Data Field Naming: Glossary "Data Field Naming" (file 1 of 33) ->
  GlossaryTerm "OverridesErasure" (strategic-products-vocabulary.md,
  the last file that creates anything; the classify-* files after it
  only classify)
- Data Governance Program: Glossary "Employee Glossary" (18 of 19) ->
  InformationSupplyChain "Data Subject Rights Information Supply
  Chain" (strategic-information-supply-chains.md, 19 of 19)
- Martyn's Law: Regulation (3 of 4) -> CollectionFolder "Martyn's Law
  (UK)" (collections.md, 4 of 4)
- Local Dashboards: WorkItemList (1 of 8) -> Agreement "Marketing
  Subscription - C360" (the last file that writes to Egeria; reports
  and dashboard sheets go to a local file)

Sustainability is unchanged: its second file only adds links, so file 1
is the last file that creates an element.

Docs: canary guidance in data-initialization-manifests.md, plus the
shipped examples and the user guide's example.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
@mandy-chessell
mandy-chessell merged commit b1540c2 into odpi:main Oct 9, 2026
7 checks passed
mandy-chessell pushed a commit that referenced this pull request Oct 9, 2026
@dwolfson
chore(release): prepare 6.2 — Egeria 6.2 images, release notes, chang…
4fc92ae 
…elog

- Pin quickstart (docker.io) and freshstart (quay.io) Dockerfile-egeria-platform
  to odpi/egeria-platform:6.2@sha256:8ea145c7…, the published Egeria 6.2 image
  (same digest on both registries; the image's jar is
  omag-server-platform-6.2.jar). --refresh-platform still re-pins to latest.
- RELEASE_NOTES.md: Egeria Workspaces 6.2, v6.1..HEAD (686 commits) plus #618.
- CHANGELOG.md: fold [Unreleased] and the early [6.2.0] entry into one [6.2]
  entry, add #618 and the image pin, and compare against tag v6.2 (this
  repo's tag style), not v6.2.0.
- .github/release-template.md: alignment is Egeria v6.2 / pyegeria 6.2.0.

Signed-off-by: Dan Wolfson <dan.wolfson@pdr-associates.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mandy-chessell mandy-chessell mandy-chessell approved these changes

Assignees

@dwolfson dwolfson

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dwolfson @mandy-chessell