Skip to content

docs(benchmark): rebase LongMemEval docs onto current main (supersedes #332, CONFLICTING resolve) - #407

Closed
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:docs/rebase-332-lme-benchmark-conflict
Closed

docs(benchmark): rebase LongMemEval docs onto current main (supersedes #332, CONFLICTING resolve)#407
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:docs/rebase-332-lme-benchmark-conflict

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

背景

PR #332 (docs(benchmark): document LongMemEval context_answer/answer_judge steps) 创建于 2026-07-08,距今 23 天,状态为 MERGEABLE=CONFLICTING

根因#332 修改了 7 个文件,其中 4 个(docs/en/_toc.ymldocs/en/index.mddocs/zh/_toc.ymldocs/zh/index.md在当前 main 分支已不存在——项目文档结构从 Jupyter-Book 的 _toc + index 卡片系统重构为 flat Markdown 目录树。因此 git merge 无法定位 base,造成 conflict。

本 PR 放弃 4 个已不存在文件的改动,仅把 #332 真正有价值的 3 部分核心内容(2 篇完整 benchmark 页面 + 1 行 quick_start 链接)移植到当前 main 基线,完成 #332 的功能,同时解决 conflict。

改动内容

继承自 #332,措辞未变:

  1. docs/en/lme_benchmark.md(新建):143 行完整英文文档

    • 两个 Step 概览表 (context_answer_step / answer_judge_step)
    • context_answer_step 必填输入字段与 output 结构
    • answer_judge_step 4 类 question_type → judge prompt 映射表 + 归一化规则(- space_,大小写不敏感)+ yes/no 正则提取
    • lme_one_question Job 样例 YAML 与 CLI 调用示例
    • 4 条 Benchmark 作者注意事项(ValueError 语义 / harness 侧归一化 / answer_judgement 评分字段 / metadata 持久化)
  2. docs/zh/lme_benchmark.md(新建):136 行中文翻译,与 en 版严格对齐

适配当前结构:

  1. docs/zh/quick_start.md:在 auto_memory/auto_resource/auto_dream/proactive 链接段落后,保持原有 docs(benchmark): document LongMemEval context_answer/answer_judge steps #332 位置,加入指向 ./lme_benchmark.md 的 2 行说明
  2. docs/en/quick_start.mddocs(benchmark): document LongMemEval context_answer/answer_judge steps #332 没做)本 PR 补齐英文镜像链接,保持中/英 quick_start 对称,避免双语漂移

明确不做(边界):

验证方式

  1. Pre-commit: pre_commit run --files <4 个文件> → 12/12 hooks 通过
  2. Unit tests: pytest tests/unit -x -q858 passed 零回归
  3. 链接检查
    • docs/zh/quick_start.md:178./lme_benchmark.md 存在 ✅
    • docs/en/quick_start.md:184./lme_benchmark.md 存在 ✅
    • 两页内部引用(context_answer_step / answer_judge_step 名称)与 reme/config/jinli_lme.yaml + 实际代码实现一致(对照 feat(benchmark): add lme_one_question job to jinli_lme.yaml #333 的 test_config_parser test 验证)
  4. 手动语义检查answer_judge_step 正则 ^\s*(yes|no)\b 大小写不敏感、归一化规则、4 个 question_type 名称均与 feat(benchmark): add lme_one_question job to jinli_lme.yaml #333 的测试 expectations 一致

影响范围

Checklist

…sedes #332)

The old PR #332 has been CONFLICTING for ~23 days because its doc layout
changes (docs/en/_toc.yml, docs/en/index.md, docs/zh/_toc.yml,
docs/zh/index.md) referred to files that no longer exist in current main
(the docs layout was refactored to a flat Markdown tree between the time
#332 was authored and today's HEAD). Rather than try to rebase those
obsolete file edits onto a layout that doesn't have them, this PR ports
the genuinely useful content from #332 (the two language-specific
benchmark pages) onto today's main and drops the 4 non-existent file
edits entirely.

Carried over, unchanged in wording:
- docs/en/lme_benchmark.md  (new) — 143 lines covering context_answer_step
  + answer_judge_step inputs/outputs, judge-prompt selection table, job
  wiring example with lme_one_question, and the 4 "notes for benchmark
  authors" footguns
- docs/zh/lme_benchmark.md  (new) — same content in Chinese

Adapted to current layout:
- docs/zh/quick_start.md — still adds the 2-line pointer to the benchmark
  page in the same logical location (right after the auto_* / proactive
  paragraph), but now the link works because the target file is actually
  in docs/zh/
- docs/en/quick_start.md — added the English mirror of the zh line so
  the bilingual quick starts stay symmetric

Explicitly NOT carried over from #332 (since those files don't exist):
- docs/en/_toc.yml, docs/en/index.md Benchmark section + LME card
- docs/zh/_toc.yml, docs/zh/index.md 评测 section + LME card
  (Jupyter-Book toc/index layout is gone; flat docs don't need them.)

If maintainers later want the Benchmark nav group back, that is best
introduced alongside whatever flat-docs navigation scheme replaces the
old Jupyter-Book toc, not in this PR which is scoped to conflict
resolution of the original benchmark prose.

Supersedes #332. Relates to #354 (complete LongMemEval evaluation).
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@jinliyl
jinliyl requested review from ployts and xyf2020 and removed request for xyf2020 July 31, 2026 02:10
@RerankerGuo RerankerGuo closed this by deleting the head repository Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant