Add POC Validator agent sample - #1947
Conversation
Reviews a proposed AWS architecture against segment and industry rules, checks that the services actually connect, prices it, scores the Scope of Work, and recommends further reading from AWS sources only — then, optionally, runs behind a small web front end so a reviewer can upload a real SOW and hand back a view-limited, expiring share link. AgentCore features: Runtime, Memory, Gateway, Identity, Policy Engine, Evaluations, Observability. The deterministic core (validation, pricing, SOW scoring, recommendations) never asks a model to compute a number a reviewer would take at face value — only diagram reading and SOW-prose banding go through the model, and both degrade gracefully to a deterministic floor if the model call fails. Documented in docs/decisions/ (8 ADRs) and self-assessed against the repo's own use-case rubric in docs/self-assessment.md. Deployed and verified end to end in a real AWS account before submission: agentcore deploy, agentcore invoke, and direct InvokeAgentRuntime calls all confirmed working, including the graceful-degradation path when this account's Bedrock Marketplace payment gate blocks model calls. 67 tests, under a second, no network, no AWS account required — run from a fresh clone as part of preparing this submission.
A full teardown-and-redeploy while preparing this PR reconfirmed that
CreateEvaluator fails in this account for the evaluator grading model —
this time with a different error ("Role does not have access for model")
than the one that led to picking this model ID originally, pointing at
the account's broader Bedrock Marketplace restriction rather than a
per-model quirk.
The README and self-assessment previously implied evaluators deploy by
default; deploy.sh has never actually shipped them successfully. Fixed:
- agentcore.json.template now ships with evaluators: [] to match what
./deploy.sh actually deploys, so a fresh user doesn't hit the same
failure on their first try
- README's services table, Known Limitations, and setup comment now
state this plainly instead of implying evaluators are live
- self-assessment.md's feature audit and failure log updated with the
new evidence
- evaluators-stage2.json's note updated with the latest error, for
whoever deploys into an account without this restriction
Also fixed: a Layout-section reference to "ADR 0011" that read as if
this sample had 11+ ADRs of its own (it mirrors ADR 0011 in the
event-driven-claims-agent sample, now stated explicitly).
67 tests still pass; architecture.png re-verified accurate (it was
never evaluator-specific, so no diagram change needed).
|
Follow-up commit: a word-for-word review before merge caught that the README/self-assessment implied the two custom Evaluators deploy by default — they never have, in this account ( |
A real, from-scratch fresh-clone verification (not just re-running tests against my already-populated working directory, which was the blind spot) found that the repo's root .gitignore was silently excluding files this sample genuinely needs, none of it caught by the earlier "fresh clone" test because that test ran in a directory where the files were already physically present on disk regardless of git-tracking status: - agentcore/cdk/lib/ and web/cdk/lib/ — matched the blanket `lib/` rule. web/cdk/lib/web-stack.ts is the actual CDK stack definition; without it `cdk synth`/`cdk deploy` have nothing to deploy. - app/pocvalidator/Dockerfile and agentcore/mcp-targets/aws-documentation/Dockerfile — matched `**/Dockerfile`. Without the first, `agentcore deploy` cannot build the Runtime image at all. - .env.example — matched `.env.*`. - .dockerignore — matched the blanket rule. - data/samples/diagrams/simple-webapp.drawio — matched `*.drawio` (that rule exists to keep rendered-PNG-only diagrams out of the repo; this file is a parser test fixture, not an architecture diagram, and core/diagrams.py's draw.io parsing is core functionality this sample demonstrates). Fixed with explicit negation entries in the root .gitignore, following the exact precedent already set for other samples in this same file (e.g. the `lib/` and `**/Dockerfile` blocks already carry negations for visa-b2b-account-payable-agent and event-driven-claims-agent respectively). Verified for real this time: fresh `git clone` of this branch into a clean directory, `python -m pytest tests/ -q` from that clean clone (not the working copy), `cdk synth` against the now-present web-stack.ts.
|
Second follow-up: found something more serious doing a genuine fresh-clone test (not just re-running tests in my already-populated working directory, which was hiding this). The repo's root |
Adds AgentCore Memory's third built-in strategy alongside the existing SEMANTIC and SUMMARIZATION ones, so a repeat reviewer's inferred preferences (region, segment, industry they tend to submit) persist across sessions instead of being re-derived every call. Deployed and verified live: agentcore deploy → UPDATE_COMPLETE, Memory now reports SEMANTIC + SUMMARIZATION + USER_PREFERENCE, a real end-to-end invocation through the web layer succeeded post-deploy. The part that actually took engineering, not just a config entry: every Memory namespace here is keyed by actor_id, and the web layer's Lambda had every visitor hardcoded to "web-demo" — meaning USER_PREFERENCE would have silently blended every visitor's inferred preferences into one meaningless composite rather than doing nothing. Fixed by generating a per-browser id client-side (crypto.getRandomValues, kept in localStorage) and threading it through as user_id, so the feature means something for the one entry point most people will actually use. Documented in ADR 0009, including why this is a deliberate, disclosed per-browser identity rather than a login system. Also fixes a docs/ directory duplication left over from an earlier sync (a `cp -r` had nested a stale docs/docs/ copy inside docs/ in the local working tree — never part of this branch's history, caught before it could become one). 67 tests still pass.
|
Follow-up feature: added AgentCore Memory's USER_PREFERENCE strategy (third strategy alongside the existing SEMANTIC + SUMMARIZATION) so a repeat reviewer's inferred preferences persist across sessions. Deployed and verified live (Memory now reports all three strategies, real invocation succeeded post-deploy). The part worth calling out: every Memory namespace here is keyed by actor_id, and the web layer's Lambda had every visitor hardcoded to the same fake identity — which would have made USER_PREFERENCE silently blend everyone's preferences together. Fixed with a per-browser client-side id (localStorage) instead of a login system, documented in ADR 0009 including why that's a deliberate, disclosed simplification rather than real auth. |
… shared FAQ Knowledge Base Two new optional phases, both real AgentCore usage, both verified directly against the deployed AWS account rather than assumed from documentation: - Phase 6a (tools/what_if_pricing.py): a reviewer's plain-language pricing question is turned into a Haiku-authored compute(lines) function, executed in AgentCore Code Interpreter's AWS-managed sandbox (aws.codeinterpreter.v1) against the real cost line items — never a number the model just states. See ADR 0010. The sandbox call was verified directly, independent of the code-authoring model call, which is expected to degrade under this account's standing Bedrock Marketplace restriction (same as SOW grading). Caught and fixed one real bug in the process: InvokeCodeInterpreter's response is an EventStream, not the flat dict the API reference implies. - Phase 6b (tools/faq_search.py): recurring review findings (HIPAA/VPC, RDS storage class, Multi-AZ, WAF, and more — agentcore/faq/*.md) are grounded in a curated AgentCore Knowledge Base (FMKB), queried with plain bedrock-agent-runtime:Retrieve rather than RetrieveAndGenerate, so this path carries no generation-model dependency at all. See ADR 0011 for why a shared Knowledge Base fits this better than a per-actor Memory namespace. Verified end-to-end, including catching a second real bug: AgentCore-managed knowledge bases reject vectorSearchConfiguration and require managedSearchConfiguration instead. Both features needed manual IAM grants and, for the knowledge base, an explicit env var — confirmed by inspecting the deployed stack directly that the CDK L3 construct does not auto-provision either for a knowledgeBases[] resource the way it does for Memory and Gateway. scripts/grant_code_interpreter_access.sh and scripts/grant_faq_knowledge_base_access.sh close those gaps and are now wired into deploy.sh. Also: 2 new tests (69 total), README/ARCHITECTURE/self-assessment updated, AgentCore feature count 7->8 (Code Interpreter is one of the rubric's 13; Knowledge Base is documented but not claimed as a rubric point since it isn't one of the named 13).
|
Pushed two additional AgentCore features, both deployed and verified against a real account before this comment: Code Interpreter — what-if pricing (Phase 6a, optional). A reviewer's plain-language cost question ("what if we dropped Multi-AZ") is turned into a Haiku-authored Shared FAQ search — Knowledge Base / FMKB (Phase 6b, optional). Recurring architecture-review findings (HIPAA/VPC, RDS storage class, Multi-AZ, WAF, and more) are grounded in a curated Both needed manual IAM grants (and, for the knowledge base, an explicit env var) — confirmed by inspecting the deployed stack that the CDK L3 construct doesn't auto-provision either the way it does for Memory and Gateway. Test suite is up to 69 (2 new), and the self-assessment's AgentCore feature count moved from 7 to 8 (Code Interpreter is one of the rubric's 13 named features; the Knowledge Base addition is documented but not claimed as a rubric point since it isn't one of the 13). |
…IAM grant and env var deploy.sh never called scripts/grant_faq_knowledge_base_access.sh — a fresh deploy would provision the knowledge base with no bedrock:Retrieve grant and no FAQ_KNOWLEDGE_BASE_ID env var, so FAQ search would silently stay "unavailable" forever until an operator read ADR 0011 and ran the script by hand. The script's own docstring even claimed it wrote the env var when it only printed instructions to do so manually. Fixed both: the script now actually patches agentcore.json's runtime envVars with the discovered knowledge base ID (idempotent — a second run against an already-correct file is a no-op, verified by removing the entry and re-running), and deploy.sh calls it and redeploys once more only when a patch actually happened. Also: README's `agentcore invoke` example now shows what_if_question and faq_query, and docs/ARCHITECTURE.md's cost table gets rows for Code Interpreter and Knowledge Base (real, sourced consumption-based pricing models — no fabricated per-GB/per-retrieval numbers, since AWS's own announcement points to the pricing page for current rates rather than publishing them inline).
…nd cleanup Security review (bandit, pip-audit, npm audit, secrets scan, IAM review): - Fix XXE/entity-expansion exposure in draw.io diagram parsing (core/diagrams.py): swap xml.etree.ElementTree for defusedxml.ElementTree, a drop-in-compatible parser, since the input is an uploaded file, not internal data. - Fix non-constant-time comparison of the web layer's demo-key header (web/lambda/handler.py): use hmac.compare_digest instead of !=. - Mitigate a HIGH-severity brace-expansion advisory bundled inside aws-cdk-lib (both CDK apps) via an npm overrides pin, and document why one copy remains unreachable: it ships as a bundled dependency of aws-cdk-lib itself, confirmed via its bundleDependencies field and by testing the latest published aws-cdk-lib release, which still bundles an affected version. Full detail in README's Known Limitations. - No wildcard IAM resources found anywhere in scope; no hardcoded credentials or account identifiers in tracked files. Test coverage: app/pocvalidator/core/*.py — the deterministic, offline testable core — now at 100% line coverage (was 72%), 106 tests (was 69). main.py, memory/session.py, and tools/*.py remain covered by live-deployment verification rather than mocks, consistent with this sample's existing no-mocking-AWS convention. Added a CDK synthesis test suite for web/cdk/lib/web-stack.ts, which previously had none, covering IAM scoping, the CloudFront dual-auth permission fix, and CloudFront behavior ordering. Cleanup: removed agentcore/mcp-defs.json, an empty, unreferenced stub file present since the initial commit. Regenerated architecture.png to include Code Interpreter and the Knowledge Base. Revised ADR 0010 and ADR 0011 for a more direct, declarative style consistent with the sample's earlier ADRs.
…ty scanners, standalone-repo hygiene - Bump aws-cdk-lib ~2.261.0 -> 2.264.0 (exact) in both CDK apps; all reachable brace-expansion copies now >=5.0.9 via overrides (bundled 5.0.8 remains, documented in Known Limitations) - Raise Python dependency floors to verified versions (bedrock-agentcore 1.21, strands-agents 1.51, mcp 1.29, botocore 1.43.69, otel-distro 0.19) - Add .gitlab-ci.yml: pytest + pip-audit + both jest suites + GitLab SAST, Secret Detection (historic), Dependency Scanning, IaC scanning - Add Apache-2.0 LICENSE and docs/CI_AND_SECURITY.md; fix README links that pointed outside the repo; test count 69 -> 106 - Convention test no longer scans developer-local virtualenvs
… CI runs python-lint.yml runs ruff check AND ruff format --check on every changed file; js-lint.yml runs eslint and prettier --check. Verified both directly rather than assuming clean, and found real gaps: - ruff format --check failed on 9 files. Applied ruff format, then fixed every ruff check finding surfaced in files this PR touches (main.py, core/chaining.py, core/diagrams.py, core/models.py, core/rules.py): two unused imports removed, one condition inlined (SIM103), four implicit string concatenations parenthesized (ISC004), and six blind `except Exception` blocks annotated with `# noqa: BLE001` plus a one-line reason — matching the pattern already used in tools/what_if_pricing.py and tools/faq_search.py, since these are a deliberate graceful-degradation design choice, not an oversight. - web/cdk had no .prettierrc (agentcore/cdk does); copied it over and ran Prettier across web/cdk's TypeScript so js-lint.yml's --check step passes instead of just not-failing via continue-on-error. All findings were pre-existing in files this PR's diff touches, not introduced by earlier commits in this branch — confirmed by diffing against a stash of the pre-existing tree before making any of these fixes.
1 participant
Summary
Adds a new sample under
02-use-cases/02-workflow-automation-agents/poc-validator-agent/.Reviews a proposed AWS architecture against segment and industry rules, checks that the services actually connect, prices it, scores the Scope of Work against fixed criteria, and recommends further reading from AWS sources only — then, optionally, runs behind a small web front end so a reviewer can upload a real SOW and hand back a view-limited, expiring share link, without a login for the recipient. Two further optional phases let a reviewer ask a plain-language what-if pricing question, answered by model-authored code executed in a real sandbox rather than a number the model states directly, and search a curated FAQ of recurring findings by vector retrieval.
AgentCore features used: Runtime, Memory (SEMANTIC + SUMMARIZATION + USER_PREFERENCE, split explicitly by short-term/long-term scope), Gateway (real Lambda-backed AWS Documentation MCP target), Identity (M2M OAuth via Cognito), Policy Engine (Cedar,
ENFORCEmode), Evaluations, Observability, Code Interpreter (AWS-managed sandbox, what-if pricing). A Knowledge Base (FMKB) is also used for shared FAQ search; it is documented but not claimed as a rubric point, since Knowledge Base is not one of the rubric's 13 named features.Design principle carried through the whole sample: the model is used only where judgement is needed — reading a diagram, banding SOW prose, authoring a what-if calculation. Validation, pricing, and recommendations are deterministic Python, so nothing a reviewer would take at face value is generated. If a model call fails, SOW scoring and what-if pricing both degrade to a deterministic floor or a clear "unavailable" result and say so, rather than failing the review — observed live against a real deployment, not just unit-tested.
Eleven ADRs in
docs/decisions/capture the reasoning behind each of these choices.Verified before submission
InvokeCodeInterpreter's response is an event stream, not the flat dict the API reference implies; AgentCore-managed knowledge bases requiremanagedSearchConfiguration, not thevectorSearchConfigurationshown in standard Bedrock Knowledge Base tutorials. Both fixed and re-verified.bandit,pip-audit,npm audit, a repository-wide secrets scan, and a manual IAM review of every grant script and CDK construct. Two findings fixed — an XXE exposure in diagram parsing (nowdefusedxml), a non-constant-time credential comparison (nowhmac.compare_digest). One dependency advisory remains, documented rather than hidden: a HIGH-severitybrace-expansionfinding bundled insideaws-cdk-libitself, unreachable bynpm overridesbecause it ships as a bundled dependency — confirmed againstaws-cdk-lib'sbundleDependenciesfield and against its latest published release. No wildcard IAM resources anywhere in scope; no hardcoded credentials or account identifiers in any tracked file.python -m pytest tests/ -q) — 100% line coverage onapp/pocvalidator/core/*.py, the deterministic, offline-testable core (was 72%).main.py,memory/session.py, and the AWS-calling tool modules are covered by live-deployment verification rather than mocks, consistent with this sample's existing no-mocking convention. A CDK synthesis test suite was added forweb/cdk/lib/web-stack.ts, which previously had none.ruff checkagainst this repo's ownpyproject.toml: clean on every new/changed file.02-use-cases/use-case-assessment.md's own rubric indocs/self-assessment.md: 17 (UPDATEtier).architecture.png) regenerated with thediagramsPython package to include Code Interpreter and the Knowledge Base.Test plan
python -m pytest tests/ -q— 106 passed, from a fresh clone of this branch; 100% coverage onapp/pocvalidator/core/*.pyruff check 02-use-cases/02-workflow-automation-agents/poc-validator-agent/against the repo'spyproject.tomlagentcore validate→Valid, against a real deployment in a separate AWS accountagentcore invoke/InvokeAgentRuntimecalls covering every phase, including both optional ones, and the graceful model-fallback path firing for realStartCodeInterpreterSession/InvokeCodeInterpreter,bedrock-agent-runtime:Retrieve) against the deployed resources, independent of the full agent flowbandit,pip-audit,npm audit, and a manual IAM/secrets review across the full diffnpx jestin bothagentcore/cdk/andweb/cdk/python3 diagrams.pyregeneratesarchitecture.pngcleanly