Skip to content

Add EC2 capacity provider observability sample - #1945

Open
balancin wants to merge 5 commits into
awslabs:mainfrom
balancin:sample/ec2-capacity-provider-observability
Open

Add EC2 capacity provider observability sample#1945
balancin wants to merge 5 commits into
awslabs:mainfrom
balancin:sample/ec2-capacity-provider-observability

Conversation

@balancin

Copy link
Copy Markdown
Contributor

What this adds

A new sample: a Strands agent on an AWS-managed EC2 instance (Instances compute type / capacity provider), behind an AgentCore Gateway, with end-to-end OpenTelemetry. The client's W3C traceparent becomes the parent of the agent's spans, so a single trace spans client → Gateway → runtime → EC2 → Bedrock, and gen_ai.* usage lands in the GenAI Observability screen.

Path: 01-features/02-host-your-agent/01-runtime/01-hosting-agents/10-ec2-capacity-provider-observability/

Why

It sits between two existing samples and fills the gap between them:

  • 08-http-ec2-capacity-provider — the capacity provider, but no observability
  • 04-observability-with-strands — observability, but serverless

This is the intersection: observability of a capacity provider.

What deploy.py sets up (idempotent)

IAM (3 roles) → private subnet with no public IP and a zero-ingress SG → the 7 VPC endpoints a private subnet needs → the image (built on CodeBuild, no local Docker) → CapacityProvider → runtime → Gateway → Transaction Search. invoke.py exercises the Gateway and the direct path and verifies the trace survived; cleanup.py removes everything in the order the service requires.

Conventions followed

  • Same file layout as the neighbouring samples; image built via CodeBuild with the Dockerfile inlined in deploy.py (no versioned Dockerfile, per repo .gitignore).
  • Plain public boto3 (≥ 1.43.66) — no vendored SDK.
  • Region has no silent default (falls back to the profile); refuses to run rather than launch EC2 somewhere you did not choose.

Validation

Validated end to end on a real account in us-east-2 and us-west-2: cold start 57–76 s, warm 2–3 s, every trace matched, EC2 with no public IP (IMDS public-ipv4 → 404), and the full deploy → invoke → cleanup cycle leaves nothing running.

The README documents the traps hit in practice, each with its fix: the OTLP region needs a single source (ADOT signs SigV4 with the region parsed from the endpoint, so a baked-in region wins — deploy.py sets it per region); every VPC endpoint is load-bearing and a missing xray/bedrock-runtime one fails silently; the Gateway needs protocolType omitted and serverProtocol=HTTP; one deployment per VPC (interface endpoints take one subnet per AZ); and instanceProfileArn is omitted so there is no shared fixed-name IAM role.

A Strands agent on an AWS-managed EC2 instance (Instances compute type /
capacity provider), behind an AgentCore Gateway, with end-to-end OpenTelemetry:
the client's W3C traceparent becomes the parent of the agent's spans, so a single
trace spans client -> Gateway -> runtime -> EC2 -> Bedrock, and gen_ai.* usage
lands in the GenAI Observability screen.

This sits between two existing samples: 08-http-ec2-capacity-provider (the CP, no
observability) and 04-observability-with-strands (observability, serverless). It
is the intersection: observability OF a capacity provider.

What it sets up (deploy.py, idempotent): three IAM roles, a private subnet with no
public IP and a zero-ingress SG, the 7 VPC endpoints a private subnet needs, the
image (built on CodeBuild, no local Docker), the CapacityProvider, the runtime,
the Gateway, and Transaction Search. invoke.py exercises both the Gateway and the
direct path and verifies the trace survived; cleanup.py removes everything in the
order the service requires.

Follows the repo conventions: same file layout as the neighbouring samples, image
built via CodeBuild with the Dockerfile inlined in deploy.py (no versioned
Dockerfile), plain public boto3 (>= 1.43.66).

Validated end to end in us-east-2 and us-west-2 on a real account: cold start
57-76s, warm 2-3s, every trace matched, EC2 with no public IP (IMDS public-ipv4
-> 404), and the full deploy -> invoke -> cleanup cycle leaves nothing running.

The README documents the traps hit in practice, each with the fix: the OTLP
region must have a single source (ADOT signs SigV4 with the region parsed from
the endpoint, so a baked-in region wins — deploy.py sets it per region on the
runtime); every VPC endpoint is load-bearing and a missing xray/bedrock-runtime
one fails silently; the Gateway needs protocolType omitted and serverProtocol
HTTP; one deployment per VPC (interface endpoints take one subnet per AZ); and
instanceProfileArn is omitted so there is no shared fixed-name IAM role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown

Latest scan for commit: 20137fc | Updated: 2026-08-13 17:58:16 UTC

Security Scan Results

Scan Metadata

  • Project: ASH
  • Scan executed: 2026-08-13T17:58:00+00:00
  • ASH version: 3.0.0

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

Column Explanations:

Severity Levels (S/C/H/M/L/I):

  • Suppressed (S): Security findings that have been explicitly suppressed/ignored and don't affect the scanner's pass/fail status
  • Critical (C): The most severe security vulnerabilities requiring immediate remediation (e.g., SQL injection, remote code execution)
  • High (H): Serious security vulnerabilities that should be addressed promptly (e.g., authentication bypasses, privilege escalation)
  • Medium (M): Moderate security risks that should be addressed in normal development cycles (e.g., weak encryption, input validation issues)
  • Low (L): Minor security concerns with limited impact (e.g., information disclosure, weak recommendations)
  • Info (I): Informational findings for awareness with minimal security risk (e.g., code quality suggestions, best practice recommendations)

Other Columns:

  • Time: Duration taken by each scanner to complete its analysis
  • Action: Total number of actionable findings at or above the configured severity threshold that require attention

Scanner Results:

  • PASSED: Scanner found no security issues at or above the configured severity threshold - code is clean for this scanner
  • FAILED: Scanner found security vulnerabilities at or above the threshold that require attention and remediation
  • MISSING: Scanner could not run because required dependencies/tools are not installed or available
  • SKIPPED: Scanner was intentionally disabled or excluded from this scan
  • ERROR: Scanner encountered an execution error and could not complete successfully

Severity Thresholds (Thresh Column):

  • CRITICAL: Only Critical severity findings cause scanner to fail
  • HIGH: High and Critical severity findings cause scanner to fail
  • MEDIUM (MED): Medium, High, and Critical severity findings cause scanner to fail
  • LOW: Low, Medium, High, and Critical severity findings cause scanner to fail
  • ALL: Any finding of any severity level causes scanner to fail

Threshold Source: Values in parentheses indicate where the threshold is configured:

  • (g) = global: Set in the global_settings section of ASH configuration
  • (c) = config: Set in the individual scanner configuration section
  • (s) = scanner: Default threshold built into the scanner itself

Statistics calculation:

  • All statistics are calculated from the final aggregated SARIF report
  • Suppressed findings are counted separately and do not contribute to actionable findings
  • Scanner status is determined by comparing actionable findings to the threshold
Scanner S C H M L I Time Action Result Thresh
bandit 0 3 0 0 1 0 656ms 3 FAILED MED (g)
cdk-nag 0 0 0 0 0 0 5.5s 0 PASSED MED (g)
cfn-nag 0 0 0 0 0 0 34ms 0 PASSED MED (g)
checkov 0 0 0 0 0 0 4.6s 0 PASSED MED (g)
detect-secrets 0 0 0 0 0 0 916ms 0 PASSED MED (g)
grype 0 0 0 0 0 0 59.2s 0 PASSED MED (g)
npm-audit 0 0 0 0 0 0 197ms 0 PASSED MED (g)
opengrep 0 0 0 0 0 0 <1ms 0 SKIPPED MED (g)
semgrep 0 0 0 0 0 0 <1ms 0 MISSING MED (g)
syft 0 0 0 0 0 0 2.0s 0 PASSED MED (g)

Detailed Findings

Show 3 actionable findings

Finding 1: B310

  • Severity: HIGH
  • Scanner: bandit
  • Rule ID: B310
  • Location: 01-features/02-host-your-agent/01-runtime/01-hosting-agents/10-ec2-capacity-provider-observability/agent/agent.py:48-50

Description:
Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected.

Code Snippet:

try:
        with urllib.request.urlopen(req, timeout=2) as r:
            return r.read().decode()

Finding 2: B602

  • Severity: HIGH
  • Scanner: bandit
  • Rule ID: B602
  • Location: 01-features/02-host-your-agent/01-runtime/01-hosting-agents/10-ec2-capacity-provider-observability/agent/agent.py:138-141

Description:
subprocess call with shell=True identified, security issue.

Code Snippet:

# check=False on purpose: a failing command's stderr is exactly what we want to show
        r = subprocess.run(command, shell=True, capture_output=True,
                           text=True, timeout=15, check=False)
    except subprocess.TimeoutExpired:

Finding 3: B310

  • Severity: HIGH
  • Scanner: bandit
  • Rule ID: B310
  • Location: 01-features/02-host-your-agent/01-runtime/01-hosting-agents/10-ec2-capacity-provider-observability/invoke.py:130-132

Description:
Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected.

Code Snippet:

try:
        with urllib.request.urlopen(r, timeout=900) as resp:
            return json.loads(resp.read().decode()), tp, time.time() - t0, None

Report generated by Automated Security Helper (ASH) at 2026-08-13T17:57:51+00:00

Fabio Balancin and others added 4 commits August 12, 2026 16:39
The workflow lints changed files only, and it was red on every script here.
GitHub caps annotations at 10 per step, so the PR page showed 10 of the 25
the same ruff run reports locally.

13 of those 25, in deploy.py, invoke.py and agent/agent.py:

- RUF015: a single-element slice picked the main route table, so a VPC
  without one raised a bare IndexError that said nothing. Now
  next(..., None) plus an explicit error naming the VPC and why the S3
  gateway endpoint needs a main route table to attach to.
- BLE001 + S112: except Exception where the intent was a missing file or
  malformed JSON, narrowed to OSError / json.JSONDecodeError. Narrowing the
  resource-policy parse clears its S112 as well.
- PLW1510: subprocess.run without an explicit check. The run_command tool
  ignores the exit code on purpose, so check=False states it.
- F541 and I001: f-strings with no placeholders, and a missing blank line.

Four BLE001 are left on purpose, where the broad catch IS the behaviour:
the IMDS must not be able to kill the agent, and a failed EBS write is
reported in the response rather than raised. Those get a noqa with the
reason. ruff format is a separate commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The region fell back to ap-southeast-2, left over from where this sample was
first developed. A cleanup that silently targets a region you never deployed
to is worse than one that refuses to run: it either deletes something you
did not mean to touch or reports nothing to delete while the EC2 instance and
its EBS volume keep billing.

deploy.py always writes the region, so there is nothing to guess. Refuse
without it, the same way resolve_region() in deploy.py refuses to default.

Also narrows the state-file read to OSError / json.JSONDecodeError (BLE001).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
deploy.py defaulted BEDROCK_MODEL_ID to au.anthropic.claude-haiku-4-5 and
agent/agent.py to us.anthropic.claude-haiku-4-5 — two different geographic
profiles in the same sample. A geographic prefix (us./eu./apac./au.) only
resolves inside its own geography, so cloning this sample and running

    export AWS_REGION=us-east-2 && python deploy.py && python invoke.py

failed for every region outside Australia: the deploy succeeds, the instance
boots, and then every invoke returns 424/500 while CloudWatch shows

    ValidationException: The provided model identifier is invalid
    Model id: au.anthropic.claude-haiku-4-5-20251001-v1:0

The earlier end-to-end runs missed it because BEDROCK_MODEL_ID was exported in
the shell, so the default was never exercised outside ap-southeast-2.

global.anthropic.claude-haiku-4-5-20251001-v1:0 resolves in every region this
sample can run in, including sa-east-1, where global is the only prefix
available. It is also what the neighbouring 08-http-ec2-capacity-provider
sample uses (global.anthropic.claude-sonnet-4-5). The README table and the
docstring now carry the same value, with the reason.

Verified in us-east-2 after the change: deploy from scratch, image rebuilt on
CodeBuild, invoke through the Gateway and direct, both 57s cold, both traces
matching client to agent, EC2 with no public IP, EBS mounted at /mnt/data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five screenshots from a real run, in the sections that explain them:

- a single trace, from POST /invocations down to execute_tool whoami, with the
  agent answering that it is on an m6g.large with an aarch64 kernel, 2 vCPUs and
  7.7 GB in us-east-2a. That is the whole point of this sample in one frame
- the same trace as a timeline, showing the two event-loop passes the model needed
  to read a tool result, with the captured prompt and toolUse payload beside it
- spans aggregated by capability, where the model span resolves to
  AWS::Bedrock::Model and gets priced, and a tool span averages 9 ms against
  1.25 s for chat
- the capacity provider as the console renders it, which doubles as a way to check
  your own deployment: instance profile shows as "-" (the omission this README
  explains) and the lifecycle pair caps a forgotten sample at 1 hour
- model cost per trace: $0.041 for the run, ~$0.002 per trace

The account ID is redacted in the capacity provider screenshot. The intro now says
three regions rather than two, which is what was actually measured.

Follows the repo layout: an images/ directory next to the sample, like the
neighbouring samples under 01-hosting-agents.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant