Fix agent leaderboard model comparison - #109
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0bc546d738
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| <td class="num">254.4</td> | ||
| </tr> | ||
| <tr data-rank="3" data-agent="Claude Code" data-model="gpt-5.5 medium" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.29" data-steps="" data-e2e="228.4"> | ||
| <tr data-rank="3" data-agent="browser-use" data-model="gpt-5.5" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.3" data-steps="14.00" data-e2e="212.0"> |
There was a problem hiding this comment.
Do not add this browser-use result to the Agent Leaderboard as a directly comparable row: it was copied from the Model Leaderboard result originally published as a 2026-06-16 snapshot evaluated with the gpt-5.4 judge (see commit 5b94115), whereas this section explicitly states that its main-agent results use the gpt-4.1 stepwise judge. Because changing the judge can change which tasks pass, ranking the 156/210 result alongside the gpt-4.1-judged agent runs produces an invalid agent comparison; re-evaluate this run with the same judge or disclose and separate it.
Useful? React with 👍 / 👎.
| <td class="num">254.4</td> | ||
| </tr> | ||
| <tr data-rank="3" data-agent="Claude Code" data-model="gpt-5.5 medium" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.29" data-steps="" data-e2e="228.4"> | ||
| <tr data-rank="3" data-agent="browser-use" data-model="gpt-5.5" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.3" data-steps="14.00" data-e2e="212.0"> |
There was a problem hiding this comment.
Adding this sixth agent leaves the leaderboard introduction saying that the agent view compares five agents, even though the table now contains Hermes, Codex, browser-use, Claude Code, Cursor, and OpenClaw. Update the summary count so the public page accurately describes the table.
Useful? React with 👍 / 👎.
| <tr data-rank="1" data-agent="Hermes" data-model="gpt-5.5" data-browser="Lexmount" data-pass="174" data-fail="36" data-total="210" data-success="82.86" data-steps="" data-e2e="184.4"> | ||
| <td class="num">1</td> | ||
| <td><span class="cell-strong">Hermes</span></td> | ||
| <td><code class="cell-mono">gpt-5.5 medium</code></td> | ||
| <td><code class="cell-mono">gpt-5.5</code></td> |
There was a problem hiding this comment.
Keep the medium qualifier in the agent rows rather than relabeling every configuration as bare gpt-5.5. The previous Model / runtime values recorded a material inference/runtime setting (and Cursor's reported model variant), while the newly imported browser-use result only had the bare model label; erasing that distinction makes differently configured runs appear identical and removes information needed to interpret or reproduce their performance.
Useful? React with 👍 / 👎.
| <td class="num">254.4</td> | ||
| </tr> | ||
| <tr data-rank="3" data-agent="Claude Code" data-model="gpt-5.5 medium" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.29" data-steps="" data-e2e="228.4"> | ||
| <tr data-rank="3" data-agent="browser-use" data-model="gpt-5.5" data-browser="Lexmount" data-pass="156" data-fail="54" data-total="210" data-success="74.3" data-steps="14.00" data-e2e="212.0"> |
There was a problem hiding this comment.
Store this success rate at the same precision as the other agent rows. This row and Claude Code both have exactly 156 passes out of 210, but their data-success values are 74.3 and 74.29; when a user clicks the Success % header, landing/assets/script.js compares those numeric attributes and incorrectly treats the equal rates as different. Normalize the underlying values and use an explicit tie-breaker if these agents must have distinct ranks.
Useful? React with 👍 / 👎.
1 participant
Summary
Tests