Open dataset of real coding-agent tool-call failures, including Codex cases (CC BY 4.0)

[faddacake]

Hi all, I'm Charlie. I maintain a small open dataset of real coding-agent tool-call failures collected from public issues, each written up with a trajectory, expected vs. actual behavior, and a pass/partial/fail rubric: https://github.com/faddacake/agent-failure-evals (CC BY 4.0).

Some of the Codex cases, in case they're useful as regression checks:

• Codex App ignores supplied Gmail skill root and invents stale openai-bundled path #34501: the agent ignores the supplied skill root, reads a stale path, then skips the required SKILL.md.
• Cloud GitHub App reviewer fabricates make_pr follow-up commit narration with non-existent SHAs #19520: the cloud reviewer claims a commit and a make_pr follow-up that don't exist.
• Compaction stores a tool-call fragment as the summary when the model answers the summarization prompt with a tool call (no validation or retry) #55508: a tool-call fragment gets stored as the compaction summary.

I haven't re-run these against Codex myself, so they reflect what the reporters documented. If a free 5-case sample or a short mini-report on Codex failures would be useful, reply here or find me at @faddacake.