feat(ci): validate skill frontmatter on pull requests (1/4)

.claude/CLAUDE.md

@@ -76,6 +76,21 @@ metadata:
 
 The `metadata` block with `author` and `version` fields is required. Initial version should be `"1.0.0"`.
 
+### Skill Publishing Check
+
+`.github/workflows/validate-skills.yml` runs `.github/scripts/validate_skills.py` on every pull request. The rules and their reasons are in the "Skill Publishing Rules" section of [CONTRIBUTING.md](../CONTRIBUTING.md).
+
+- The rules live in `.github/scripts/skill_checks.py` as pure functions over an in-memory skill folder (`SkillTree` of `Entry(path, mode, data)`). `validate_skills.py` only reads Git objects, computes the merge base and reports. Keep rules out of the CLI so the conformance cases and unit tests exercise exactly what CI runs.
+- Every skill is checked on every PR, because skills on `main` are published as one set. Errors from any skill fail; warnings print only for skills the PR touches.
+- Skills are read from Git objects (`git ls-tree -r -z` plus `git cat-file --batch`), never the working tree, so modes are exact and a local run checks only committed changes.
+- Rule data lives in `.github/scripts/skill-rules/` and the check fails closed (exit 2) when a file there is malformed: `agent-types.json`, `vocabulary.json` (dimension values; unknown values only warn) and `conformance-cases.json`.
+- `conformance-cases.json` is a language-neutral contract: every tool that validates or publishes these skills must give each case's `expect` result. The script runs it before checking any skill and exits 2 on a mismatch. A rule change needs a matching case change in the same PR.
+- Every rule applies to every skill; nothing is exempted. A rule that existing skills don't meet yet starts as a warning (missing `metadata.summary`, dimension values outside the vocabulary) and becomes an error after a cleanup PR brings every skill into line.
+- Runtime agent types (`runtime_agent_types`): `metadata.agent_types` when set, otherwise the display values in `aws-devops-agent-skills.agent-types` mapped through `display_mapping` in `agent-types.json`, otherwise `GENERIC` (all agents), which warns. `display_mapping` must have one entry per display value in `vocabulary.json`; `null` means not agreed yet.
+- Exit codes: 0 pass (warnings allowed), 1 a rule broken, 2 could not run. Never report a 2 as the PR's fault.
+- PyYAML is the only dependency, pinned by every PyPI hash in `skill-rules/requirements.txt`; actions are pinned by commit SHA.
+- Tests: `python3 -m unittest discover -s tests -v`.
+
 ### Skill README.md Structure
 
 Each skill must have a README.md following this structure (see `skills/support-cases/README.md` as reference):

.github/scripts/skill-rules/agent-types.json

@@ -0,0 +1,33 @@
+{
+  "description": "values: the AgentType values DevOps Agent accepts for skills, which metadata.agent_types may list. display_mapping: the AgentType values each aws-devops-agent-skills.agent-types display value stands for; null means no mapping is agreed yet. A skill installs for metadata.agent_types when set, otherwise for its mapped display values, otherwise as GENERIC, which loads it for all agents.",
+  "values": [
+    "GENERIC",
+    "CHAT",
+    "INCIDENT_TRIAGE",
+    "INCIDENT_RCA",
+    "INCIDENT_MITIGATION",
+    "INCIDENT_UI",
+    "PREVENTION",
+    "GRADER",
+    "CHANGE_REVIEW",
+    "CHANGE_RELEASE",
+    "QUALITY_ASSURANCE_TESTING",
+    "RELEASE_READINESS_REVIEW",
+    "RELEASE_SHEPHERD",
+    "RELEASE_TESTING",
+    "SYSTEM_LEARNING"
+  ],
+  "display_mapping": {
+    "Chat tasks": [
+      "CHAT"
+    ],
+    "Evaluation": null,
+    "Incident RCA": [
+      "INCIDENT_RCA"
+    ],
+    "Incident Triage": [
+      "INCIDENT_TRIAGE"
+    ],
+    "Scheduled tasks": null
+  }
+}

.github/scripts/skill-rules/conformance-cases.json

@@ -0,0 +1,299 @@
+{
+  "description": "Conformance cases for the skill publishing rules. Each case is one skill folder, skills/<name>/. Every tool that validates or publishes skills from this repository must give the expect result for every case: pass means no errors (warnings are allowed), fail means at least one error. error is a fragment of this repository's own message, for validate_skills.py only. A file is given as UTF-8 text or as base64 bytes, with an optional Git mode (default 100644). validate_skills.py runs every case before it checks any skill.",
+  "cases": [
+    {
+      "id": "agent-types-unknown-value",
+      "name": "demo",
+      "expect": "fail",
+      "error": "unknown value \"PAGER\"",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  agent_types: \"CHAT, PAGER\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "author-not-github-username",
+      "name": "demo",
+      "expect": "fail",
+      "error": "GitHub usernames",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: \"octo cat\"\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "deprecated-boolean",
+      "name": "demo",
+      "expect": "fail",
+      "error": "must be text, not true or false",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  deprecated: true\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "deprecated-other-text",
+      "name": "demo",
+      "expect": "fail",
+      "error": "\"true\" or \"false\"",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  deprecated: \"yes\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "description-too-long",
+      "name": "demo",
+      "expect": "fail",
+      "error": "the limit is 1,024",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "dimension-empty-value",
+      "name": "demo",
+      "expect": "fail",
+      "error": "empty value between commas",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  aws-devops-agent-skills.aws-services: \"Amazon EKS,\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "dimension-unknown",
+      "name": "demo",
+      "expect": "fail",
+      "error": "not a known dimension",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  aws-devops-agent-skills.services: \"Amazon EKS\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "frontmatter-missing",
+      "name": "demo",
+      "expect": "fail",
+      "error": "must start with a --- line",
+      "files": {
+        "SKILL.md": {
+          "text": "# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "frontmatter-not-closed",
+      "name": "demo",
+      "expect": "fail",
+      "error": "no closing --- line",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\n"
+        }
+      }
+    },
+    {
+      "id": "metadata-missing",
+      "name": "demo",
+      "expect": "fail",
+      "error": "\"metadata\" is required",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill.\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "metadata-value-nested",
+      "name": "demo",
+      "expect": "fail",
+      "error": "must be text, not a mapping",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  extra:\n    nested: value\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "missing-skill-md",
+      "name": "demo",
+      "expect": "fail",
+      "error": "has no SKILL.md",
+      "files": {
+        "README.md": {
+          "text": "# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "name-missing",
+      "name": "demo",
+      "expect": "fail",
+      "error": "\"name\" is required",
+      "files": {
+        "SKILL.md": {
+          "text": "---\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "name-not-folder-name",
+      "name": "demo",
+      "expect": "fail",
+      "error": "but the folder is",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: other\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "name-uppercase",
+      "name": "demo",
+      "expect": "fail",
+      "error": "lowercase letters, digits and single hyphens",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: Demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "skill-md-not-utf8",
+      "name": "demo",
+      "expect": "fail",
+      "error": "not valid UTF-8",
+      "files": {
+        "SKILL.md": {
+          "base64": "LS0tCm5hbWU6IGT/bW8KLS0tCg=="
+        }
+      }
+    },
+    {
+      "id": "summary-too-long",
+      "name": "demo",
+      "expect": "fail",
+      "error": "the limit is 200",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  summary: \"sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "valid-all-optional-fields",
+      "name": "demo",
+      "expect": "pass",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: >-\n  Use this skill when testing\n  the publishing rules.\ntitle: Demo\ncompatibility: Requires nothing.\nmetadata:\n  author: \"octocat, hubot\"\n  version: \"2.10.0\"\n  summary: \"Tests the rules.\"\n  deprecated: \"true\"\n  agent_types: \"CHAT, INCIDENT_RCA\"\n  aws-devops-agent-skills.agent-types: \"Chat tasks\"\n  aws-devops-agent-skills.aws-services: \"Amazon EKS, Amazon CloudWatch\"\n  aws-devops-agent-skills.technical-domains: \"Operations\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "valid-minimal",
+      "name": "demo",
+      "expect": "pass",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "valid-unapproved-dimension-value-warns-only",
+      "name": "demo",
+      "expect": "pass",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  aws-devops-agent-skills.aws-services: \"CloudWatch\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "version-prerelease",
+      "name": "demo",
+      "expect": "fail",
+      "error": "MAJOR.MINOR.PATCH",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0-beta\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "version-two-part",
+      "name": "demo",
+      "expect": "fail",
+      "error": "MAJOR.MINOR.PATCH",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"2.6\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "version-unquoted-number",
+      "name": "demo",
+      "expect": "fail",
+      "error": "must be text, not a number",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: 2.6\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "yaml-anchor",
+      "name": "demo",
+      "expect": "fail",
+      "error": "anchors",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: &who octocat\n  version: \"1.0.0\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "yaml-duplicate-key",
+      "name": "demo",
+      "expect": "fail",
+      "error": "duplicate key \"metadata.version\"",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\n  version: \"1.0.1\"\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "yaml-flow-collection",
+      "name": "demo",
+      "expect": "fail",
+      "error": "flow collections",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: \"1.0.0\"\nallowed-tools: [read]\n---\n\n# Demo\n"
+        }
+      }
+    },
+    {
+      "id": "yaml-tag",
+      "name": "demo",
+      "expect": "fail",
+      "error": "tags",
+      "files": {
+        "SKILL.md": {
+          "text": "---\nname: demo\ndescription: Use this skill when testing the publishing rules.\nmetadata:\n  author: octocat\n  version: !!str 1.0.0\n---\n\n# Demo\n"
+        }
+      }
+    }
+  ]
+}