Cross-Region resilience for AgentCore Runtime: synthetic canary detection and a failover-routing invoker

[gilbertlep]

Cross-Region resilience for AgentCore Runtime: synthetic canary detection and a failover-routing invoker

accompanying the upcoming Disaster Recovery Resilience in Agentcore Developer Guide:

Summary

A deployable, end-to-end sample for customer-driven cross-Region failover of AgentCore Runtime. It covers the parts a customer must build themselves to fail Runtime traffic over between two Regions: a synthetic canary that health-checks the full invocation path, a CloudWatch alarm pipeline that turns the canary signal into a page, and an invoker that routes each request to a healthy Region with retry, circuit breaking, and session stickiness.

It is deployed as a warm standby across a Region pair (primary us-west-2, standby us-east-1), matching the design in the Building custom disaster recovery for Amazon Bedrock AgentCore topic in the AgentCore Developer Guide.

Problem this fills

AgentCore Runtime is Region-scoped and has no native cross-Region failover. A customer who needs to keep an agent serving through a regional impairment has to build the detection-and-routing layer themselves. The repo already covers the Memory half of cross-Region DR — 01-features/04-manage-context-of-your-agent/memory/06-production-patterns/00-multi-region-replication/ (merged in #1747) replicates LTM and STM across Regions. What is missing is the Runtime routing and health-detection half: nothing in the repo shows how to detect that a Region is unhealthy and route invocations to the standby.

A search of the repo confirms the gap: disaster recovery returns no issues, and the 03-advanced runtime folder's 13 samples (streaming, sessions, async, multi-agent, VPC, MCP, middleware, egress, async EC2 capacity provider) cover no multi-Region, failover, or health-driven routing concern.

Scope boundary (what this is and is not)

In scope: synthetic canary agent + probe, CloudWatch alarm/SNS pipeline, and a failover-routing invoker (Region selection, retry with backoff, canary-synced circuit breaker, session-route stickiness, hard fallback). Route 53 failover-record configuration for the public endpoint.

Out of scope, by design: Memory replication. This sample assumes the Memory layer is already replicated and links to the existing 00-multi-region-replication sample for it. The invoker carries the actorId and memory namespace on each call but does not itself move memory content.

Differentiation from existing samples

Existing	What it does	How this differs
00-multi-region-replication (memory, #1747)	Replicates LTM (Kinesis → Batch APIs) and STM (dual-write) across Regions	This routes Runtime invocations and detects Region health; it consumes the replicated Memory rather than producing it
13-async-ec2-capacity-provider	@app.ping HealthyBusy/Healthy for single-Region async instance reclamation	This uses an external canary across two Regions to drive cross-Region routing, not in-Region liveness

Proposed location (please confirm)

01-features/02-host-your-agent/01-runtime/03-advanced/14-cross-region-resilience/

Rationale: this is a cross-cutting Runtime concern, which is what 03-advanced holds, and 14- is the next index. It sits alongside the Runtime samples rather than under memory/, because the Memory replication sample already owns the data-layer concern. Placement remains flexible based on maintainer preference — if you would rather this live under 02-use-cases or beside the memory sample, say so and I will adjust before opening the PR.

Architecture

                    Amazon Route 53 (failover records + health checks)
                                     │
                        ┌────────────┴────────────┐
                        ▼                          ▼
              Invoker (Lambda/ECS/lib)   ── canary health (CloudWatch) ──┐
              retry · circuit breaker ·                                  │
              Region selection · session route                          │
                        │                          │                     │
        ┌───────────────┘                          └───────────┐        │
        ▼                                                       ▼        │
  us-west-2 PRIMARY (ACTIVE)                        us-east-1 STANDBY     │
  ┌───────────────────────┐                     ┌───────────────────────┐│
  │ AgentCore Runtime      │                    │ AgentCore Runtime      ││
  │ canary agent (CANARY_OK)│◄─ probe 1/min ─┐   │ canary agent           │◄┘ probe 1/min
  │ CanaryHealthy → CW      │                │   │ CanaryHealthy → CW      │
  └───────────────────────┘                 │   └───────────────────────┘
        │ CW alarm (3×60s, TreatMissingData=breaching) → SNS → page
        ▼
  AgentCore Memory (replica fed by the 00-multi-region-replication sample)

Contents and structure

Adopts the 03-advanced deploy/invoke/cleanup convention:

14-cross-region-resilience/
├── README.md              six required sections
├── agent/
│   └── canary_agent.py    minimal agent returning {"status":"CANARY_OK"} after the app path
├── canary/
│   └── canary_probe.py    Lambda probe: invoke, validate sentinel, publish CanaryHealthy
├── infra/
│   └── template.yaml       alarms, SNS, KMS-encrypted topic (per Region)
├── invoker/               deadline-bounded routing: circuit breaker, Region selection
├── deploy.py              stand up both Regions: runtime, canary, alarms, invoker
├── invoke.py              demonstrate a failover, not just a happy path
├── cleanup.py             both Regions; flags the KMS key deletion waiting period
└── requirements.txt

README outline (the six PR-checklist sections)

• Introduction — the Runtime failover problem, and the boundary with the Memory sample.
• Architecture Diagram — the diagram above, as an image.
• Prerequisites — two Regions with AgentCore Runtime; Python version; a replicated Memory (link the memory sample); IAM for InvokeAgentRuntime and CloudWatch in both Regions.
• Usage — deploy.py, then invoke.py to drive a happy path and a forced failover.
• Sample Prompts — prompts the canary agent and a demo agent answer, and the prompt used to show a failover mid-conversation.
• Clean Up — cleanup.py, with the KMS key deletion waiting period called out.

Acceptance criteria

• Deploys cleanly into a two-Region pair via deploy.py.
• invoke.py demonstrates: a healthy invocation, a forced-unhealthy primary, and a routed-to-standby invocation within the client deadline.
• Passes the repo's checkov and secrets gates.
• cleanup.py removes every resource in both Regions.

Open question for maintainers

1. Do you agree with the proposed location, or prefer a different folder?
2. Any objection to referencing (not duplicating) the 00-multi-region-replication sample for the Memory layer?