feat(ci): validate skill frontmatter on pull requests (1/4)

[miscreantmoogly]

Stacked PR 1 of 4. Draft while the proposed rules are reviewed. PRs 2–4 build on this one; review and merge in order.

Description

Adds validate-skills, a pull request check for the rules a skill must meet before it can be published. Every rule applies to every skill in skills/, not only the ones a pull request touches. A rule that existing skills don't meet yet starts as a warning, and becomes an error after a cleanup PR brings every skill into line.

This PR adds the frontmatter rules:

• plain YAML only;
• name equal to the folder name;
• limits on description, title and summary;
• metadata values as text;
• author as GitHub usernames;
• a MAJOR.MINOR.PATCH version.

It also covers the optional deprecated, agent_types and aws-devops-agent-skills.* fields, and adds three warnings:

• Missing metadata.summary. It becomes required once every skill has one.
• No agent type. A skill's agent types come from metadata.agent_types when set, and otherwise from its aws-devops-agent-skills.agent-types values through display_mapping in agent-types.json. A skill with neither would load for all agents (GENERIC), so it warns.
• Dimension values outside vocabulary.json.

Files:

• .github/scripts/skill_checks.py: the rules, as pure functions over an in-memory skill folder.
• .github/scripts/validate_skills.py: the command.
  • Reads skills from Git objects.
  • Errors from any skill fail the check; warnings show only for skills the PR touches.
  • Annotations point at the file and line.
  • Exit code 2 means the check couldn't run, and is never reported as the PR's fault.
• .github/scripts/skill-rules/:
  • agent-types.json and vocabulary.json, the rule data;
  • conformance-cases.json, language-neutral pass/fail cases that the script runs before checking any skill, and that any other validator or publisher of these skills can run too;
  • requirements.txt, PyYAML pinned by hash.
• .github/workflows/validate-skills.yml:
  • read-only, no paths filter, actions pinned by SHA;
  • runs the unit tests, then the check;
  • advisory until an admin makes validate-skills a required check.
• Version fixes: analytics-opensearch-expertise (2.6 → 2.6.1) and database-rds-devops (1.0 → 1.0.1), each with a CHANGELOG entry.
• Docs: a "Skill Publishing Rules" section in CONTRIBUTING.md, and notes in .claude/CLAUDE.md.

Like the repository's other checks, a PR runs its own copy of this one. CODEOWNERS on /.github/ (PR 4) is what stops a PR from editing the check to pass.

Result across the 34 skills on main: 0 errors, with warnings for:

• a missing summary (every skill);
• no agent type (4 skills);
• vocabulary (5 values on 3 skills).

Type of change

• Update to an existing skill, agent, or MCP server
• Documentation or infrastructure change

Testing

• python3 -m unittest discover -s tests -v: 74 tests pass. They cover the rules on in-memory skills, and the command on throwaway Git repos: exit codes, annotations, touched-only warnings, submodules, self-check failures, a missing Git object exiting 2, and newlines in values that can't start a workflow command.
• python3 .github/scripts/validate_skills.py --self-check-only: every conformance case passes.
• python3 .github/scripts/validate_skills.py --base-ref origin/main: 34 skills, 0 errors.
• pip install --require-hashes -r .github/scripts/skill-rules/requirements.txt installs into a clean venv.

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

[ams-thakkar]

Holding this one, and #125 likewise: @aadimch and @miscreantmoogly have each built the same check independently, and the two are not compatible as they stand — most pointedly, metadata.deprecated is specified oppositely (quoted "true" here, unquoted true in #125), and each documents its own form in CONTRIBUTING.md.

I have asked the two of you to agree on which set to keep.