Skip to content

Increase skills evaluation telemetry wait #1985

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rajolishruthi wants to merge 1 commit into
base: main
Choose a base branch
from
+3 −3
Open

Increase skills evaluation telemetry wait #1985

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Run one PTO prompt instead of the three built-in scenarios:
python evaluate.py \
--prompt "I am EMP-001. Can I take September 14 through September 16, 2026 off? Submit it using the pto-planning skill." \
--expected-skill pto-planning \
--wait 150
--wait 180
```

The script prints the agent response, waits for its telemetry, then runs both built-in evaluators. Change only the quoted prompt to try other PTO wording. Use `--expected-skill benefits-advisor` for a benefits prompt or `--expected-skill none` when the prompt should not load a skill.
Expand Down Expand Up @@ -157,7 +157,7 @@ Skills : benefits-advisor, pto-planning
[no-skill-control] session=skill-eval-<uuid>
Response: Here is your pay stub for January 2026 ...

[2/4] Waiting 150s for AgentCore telemetry ingestion ...
[2/4] Waiting 180s for AgentCore telemetry ingestion ...

[3/4] EvaluationClient — per-session on-demand evaluation ...

Expand Down
2 changes: 1 addition & 1 deletion ...eatures/06-observe-evaluate-optimize-your-agent/02-evaluate/skills-evaluation/evaluate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ def _parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run the built-in AgentCore skill evaluators")
parser.add_argument("--region", default=None, help="AWS region (defaults to the deployment config)")
parser.add_argument("--config", default=str(_DEFAULT_CONFIG), help="Path to the skill-enabled agent config")
parser.add_argument("--wait", type=int, default=150, help="Seconds to wait for telemetry (default: 150)")
parser.add_argument("--wait", type=int, default=180, help="Seconds to wait for telemetry (default: 180)")
parser.add_argument("--prompt", default=None, help="Run one custom prompt instead of the built-in scenarios")
parser.add_argument(
"--expected-skill",
Expand Down
Loading