[request]: DynamoDB data health inspection skill (hot keys, item size, TTL effectiveness, index utilization) #87

Select the type of tool this is related to

• Skill
• Custom agent
• MCP server
• IaC (Infrastructure as Code) template
• Other

Tell us about your request

A skill that diagnoses data-level DynamoDB health issues — the ones table-level metrics cannot explain: hot partition keys, item size distribution against the 400 KB limit, TTL effectiveness, and GSI/LSI utilization.

Which AWS service(s) is this for?

Amazon DynamoDB, Amazon CloudWatch

What problem are you trying to solve?

Existing DevOps Agent workflows can tell an operator that a DynamoDB table is throttling or consuming unexpected capacity, from control-plane metadata and CloudWatch metrics. They cannot tell them why when the cause lives in the data:

• a partition key taking a disproportionate share of traffic
• items approaching the 400 KB item size limit
• TTL enabled but silently doing nothing, because the timestamp is in milliseconds instead of epoch seconds, or stored as a String instead of a Number
• an index nobody reads that is still billed for every replicated write

That evidence only exists at the item level, and none of it appears in aggregate metrics.

The operator scenarios this covers: "our DynamoDB storage cost keeps climbing and I don't know why", "TTL is enabled but storage isn't going down", "the table throttles but consumed capacity looks low", "which GSIs are we paying for but not using", and a pre-peak review of a table's data shape.

Why it matters: measured against a corpus of 11,677 real Amazon DynamoDB support cases, these four dimensions account for 18.3% of cases (2,133), and they are the severe slice — hot-partition cases run 46% at sev1–2. Without item-level evidence these investigations stall at "the metrics look fine but it's still throttling."

How should it behave?

Triggers on throttling that persists while aggregate consumed capacity is low, unexplained storage or capacity cost growth, TTL enabled but not reclaiming storage, items possibly near 400 KB, or a request to review a table's data health, item distribution, index utilization, or schema anti-patterns.

Two phases, and the split is the point:

1. Phase A — zero data-plane cost, always runs. Control-plane reads (DescribeTable, DescribeTimeToLive, DescribeContributorInsights), Contributor Insights rule reports via cloudwatch:GetInsightRuleReport, and CloudWatch metrics including TimeToLiveDeletedItemCount and the cause-specific *KeyRangeThroughputThrottleEvents. This alone produces real findings — unused indexes, TTL deleting nothing, storage amplification.
2. Phase B — bounded, consent-gated item sampling. Only for what Phase A physically cannot see (size distribution, per-item TTL state), and only after the operator approves a stated read-unit budget. Capped at 1,000 items by default, ≤4 Scan segments, eventually-consistent, never a full-table scan. Item values are redacted — byte sizes, attribute names, and TTL timestamps only; partition keys render as digests. Sampling is refused by default against a table that is already throttling.

Safety invariants: strictly read-only, no Put/Update/Delete/Create/BatchWrite, and it never toggles a feature's enablement state in order to observe it. Contributor Insights stays authoritative for hot keys — a Scan sample is ordered by partition layout, so it measures storage distribution rather than traffic distribution and never proves a hot key on its own.

Which DevOps Agent functionalities is this relevant to?

Chat tasks, Prevention, Incident RCA

Does this need extra permissions?

Yes — beyond AIDevOpsAgentAccessPolicy:

dynamodb:Scan and dynamodb:Query are data-plane reads and would be the only non-Describe*/Get*/List* grants in cloudformation/devops-agent-skill-policies.yaml. I'd propose adding them behind a parameter defaulted to false, plus a companion parameter to scope the resource to named table ARNs rather than *.

Are you currently working around this?

Manually — an engineer pulls DescribeTable, correlates CloudWatch metrics by hand, and then writes ad-hoc Scan code to check item sizes or TTL attribute formats. That is slow, inconsistent between engineers, and the ad-hoc scans are unbounded, so they risk adding read load to a table that is already struggling.

Have you checked for existing tools?

• I've searched existing issues, skills, custom agents, and MCP servers, and this isn't already covered or extendable.
• I've confirmed DevOps Agent can't already do this natively (I tested it).

On the second box — I want to be precise rather than tick it, because my measurement is partial:

What I did measure. Agent Skill Eval runs every functional test with and without the skill. Across 13 tests, 6 require the skill (the no-skill baseline scores 0.0–0.75 on them) and 7 the baseline handles on its own. The 6 are the ones that depend on procedure rather than reasoning: the dimension taxonomy, the sampling safety contract, distinguishing a measured zero from missing data, the guard against a null threshold reading as a pass, confirming a hot key when the key-range metric is unavailable, and write amplification. Overall the skill measured a quality lift of +0.20 with cost −22.4% versus no skill.

What I could not measure cleanly. I tried a live A/B in a real Agent Space by removing the skill and re-running the same prompt. It is not a valid test: after several prior runs the agent had learned the skill's name and attempted skill_read on it, returning Skill not found before proceeding. A clean native baseline needs an Agent Space with no prior exposure.

My honest read: a capable agent handed the right structured data can reason its way to several of these conclusions. What it does not do unaided is know to collect the evidence — the Contributor Insights rule chain, correlating key-range metrics with contributors, bounding and gating a Scan, or applying calibrated thresholds consistently. That procedural knowledge, plus the safety contract around data-plane reads, is what the skill contributes.

Would you like to build it?

• I'm willing to contribute this myself (see CONTRIBUTING.md).
• I'm suggesting it for someone else to build.

Additional context

The skill is written and validated; I'll open the PR against this issue.

Validation so far, by method and what each caught:

Three findings from that work may be useful to other skill authors regardless of whether this skill is accepted:

1. A local eval passing at grade A caught none of the 16 defects the three live methods found. The dominant failure mode was missing or unaggregatable data silently reading as "healthy" — a null threshold comparison is falsy, so a check that cannot be evaluated looks like a check that passed.
2. Agents have no code execution, so per-item statistics must be aggregated in context. Asking for statistics over ~1 MB of returned items produced sub-batch counts that disagreed by 3×, and the agent correctly declined to publish them. Bounding the payload matters as much as bounding the item count.
3. A rule that recommends deleting a customer resource needs an explicit data-sufficiency guard. An unused-index rule fired at High severity on a 4-hour-old table because it conflated the queried 30-day window with 30 days of actual metric coverage.

Sample code notice: this skill is sample code, not intended for production use without additional review and testing.