Skip to content

docs(benchmark): document LongMemEval context_answer/answer_judge steps - #332

Closed
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:docs/lme-benchmark
Closed

docs(benchmark): document LongMemEval context_answer/answer_judge steps#332
RerankerGuo wants to merge 2 commits into
agentscope-ai:mainfrom
RerankerGuo:docs/lme-benchmark

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

Summary

  • Adds end-user docs for the two LME benchmark steps that landed in feat(benchmark): add lme benchmark steps #326 (context_answer_step, answer_judge_step).
  • New pages: docs/en/lme_benchmark.md and docs/zh/lme_benchmark.md; both _toc.yml get a new 'Benchmark' section that lists lme_benchmark.
  • Documents the per-step input/output contract, the question_type to judge prompt mapping, the answer_judgement normalization rules, and a self-contained job stub plus reme app invocation.
  • Calls out that the shipped jinli_lme.yaml does not currently wire the steps into a runnable job, and shows the YAML stanza a benchmark author would add on top.

Why docs only

The runtime code and example config were already merged in #326. grep -r context_answer/answer_judge on docs/ returns nothing, so this PR closes that gap without changing code or example YAML - keeps it orthogonal to whatever follow-ups are planned for #326 / #298.

Verification

  • pre-commit run --files docs/en/lme_benchmark.md docs/zh/lme_benchmark.md docs/en/_toc.yml docs/zh/_toc.yml: all hooks green.
  • Both _toc.yml files load as valid YAML and reference lme_benchmark.

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

Pushed a small follow-up commit a0cb046 on top of the existing branch:

  • docs/en/index.md and docs/zh/index.md — adds an 'LME Benchmark' card to the 'Start Reading' grid so the new page is discoverable from the docs landing without having to expand the 'Benchmark' toc section by hand.
  • docs/zh/quick_start.md — adds a one-liner right after the auto-memory/auto-resource pointer so users who follow the quick start end-to-end see the benchmark entry point as a follow-up option.

No source code or YAML touched; pre-commit run --files docs/en/index.md docs/zh/index.md docs/zh/quick_start.md stays green.

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

Superseded by #407, which rebases the same LongMemEval documentation onto the current main and is conflict-free. Closing this older conflicting PR to keep the review queue focused.

@RerankerGuo RerankerGuo closed this Aug 6, 2026
@RerankerGuo

Copy link
Copy Markdown
Contributor Author

Closing this older conflicting PR because #407 carries the still-applicable LongMemEval documentation on the current repository layout and explicitly supersedes this branch. This avoids asking maintainers to review two versions of the same documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant