docs(benchmark): rebase LongMemEval docs onto current main (supersedes #332, CONFLICTING resolve) - #407
Closed
RerankerGuo wants to merge 1 commit into
Closed
Conversation
…sedes #332) The old PR #332 has been CONFLICTING for ~23 days because its doc layout changes (docs/en/_toc.yml, docs/en/index.md, docs/zh/_toc.yml, docs/zh/index.md) referred to files that no longer exist in current main (the docs layout was refactored to a flat Markdown tree between the time #332 was authored and today's HEAD). Rather than try to rebase those obsolete file edits onto a layout that doesn't have them, this PR ports the genuinely useful content from #332 (the two language-specific benchmark pages) onto today's main and drops the 4 non-existent file edits entirely. Carried over, unchanged in wording: - docs/en/lme_benchmark.md (new) — 143 lines covering context_answer_step + answer_judge_step inputs/outputs, judge-prompt selection table, job wiring example with lme_one_question, and the 4 "notes for benchmark authors" footguns - docs/zh/lme_benchmark.md (new) — same content in Chinese Adapted to current layout: - docs/zh/quick_start.md — still adds the 2-line pointer to the benchmark page in the same logical location (right after the auto_* / proactive paragraph), but now the link works because the target file is actually in docs/zh/ - docs/en/quick_start.md — added the English mirror of the zh line so the bilingual quick starts stay symmetric Explicitly NOT carried over from #332 (since those files don't exist): - docs/en/_toc.yml, docs/en/index.md Benchmark section + LME card - docs/zh/_toc.yml, docs/zh/index.md 评测 section + LME card (Jupyter-Book toc/index layout is gone; flat docs don't need them.) If maintainers later want the Benchmark nav group back, that is best introduced alongside whatever flat-docs navigation scheme replaces the old Jupyter-Book toc, not in this PR which is scoped to conflict resolution of the original benchmark prose. Supersedes #332. Relates to #354 (complete LongMemEval evaluation).
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
jinliyl
requested review from
ployts and
xyf2020
and removed request for
xyf2020
July 31, 2026 02:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
PR #332 (docs(benchmark): document LongMemEval context_answer/answer_judge steps) 创建于 2026-07-08,距今 23 天,状态为
MERGEABLE=CONFLICTING。根因:#332 修改了 7 个文件,其中 4 个(
docs/en/_toc.yml、docs/en/index.md、docs/zh/_toc.yml、docs/zh/index.md)在当前 main 分支已不存在——项目文档结构从 Jupyter-Book 的 _toc + index 卡片系统重构为 flat Markdown 目录树。因此 git merge 无法定位 base,造成 conflict。本 PR 放弃 4 个已不存在文件的改动,仅把 #332 真正有价值的 3 部分核心内容(2 篇完整 benchmark 页面 + 1 行 quick_start 链接)移植到当前 main 基线,完成 #332 的功能,同时解决 conflict。
改动内容
继承自 #332,措辞未变:
docs/en/lme_benchmark.md(新建):143 行完整英文文档-space→_,大小写不敏感)+yes/no正则提取lme_one_questionJob 样例 YAML 与 CLI 调用示例docs/zh/lme_benchmark.md(新建):136 行中文翻译,与 en 版严格对齐适配当前结构:
docs/zh/quick_start.md:在auto_memory/auto_resource/auto_dream/proactive链接段落后,保持原有 docs(benchmark): document LongMemEval context_answer/answer_judge steps #332 位置,加入指向./lme_benchmark.md的 2 行说明docs/en/quick_start.md(docs(benchmark): document LongMemEval context_answer/answer_judge steps #332 没做):本 PR 补齐英文镜像链接,保持中/英 quick_start 对称,避免双语漂移明确不做(边界):
docs/{en,zh}/_toc.yml+index.md:这 4 个文件已从项目结构中移除。若 maintainer 后续决定在新的 flat-docs 导航体系中重开"评测"栏目,应作为独立 PR 处理,不应与本次 conflict 解冲突 PR 混杂验证方式
pre_commit run --files <4 个文件>→ 12/12 hooks 通过pytest tests/unit -x -q→ 858 passed 零回归docs/zh/quick_start.md:178→./lme_benchmark.md存在 ✅docs/en/quick_start.md:184→./lme_benchmark.md存在 ✅context_answer_step/answer_judge_step名称)与 reme/config/jinli_lme.yaml + 实际代码实现一致(对照 feat(benchmark): add lme_one_question job to jinli_lme.yaml #333 的 test_config_parser test 验证)answer_judge_step正则^\s*(yes|no)\b大小写不敏感、归一化规则、4 个 question_type 名称均与 feat(benchmark): add lme_one_question job to jinli_lme.yaml #333 的测试 expectations 一致影响范围
Checklist