feat(benchmark): add lme_one_question job to jinli_lme.yaml - #333
feat(benchmark): add lme_one_question job to jinli_lme.yaml#333RerankerGuo wants to merge 2 commits into
Conversation
|
Pushed a small follow-up commit
So if the future of |
aa0538a to
ea7c18d
Compare
ea7c18d to
3b15969
Compare
|
Closing as superseded by the benchmark restructuring merged in #375. Current |
|
Closing as obsolete after the LongMemEval benchmark restructuring in #375. The target |
Summary
lme_one_questiontoreme/config/jinli_lme.yaml.context_answer_step,answer_judge_step) so the LME harness can POST one HTTP request per question and get both the direct-context answer and the LLM-judge verdict back in one round trip.query,session_context,current_date,agent_answer,golden_answer,question_type) are declared as required parameters with per-field descriptions, matching the input contract of the two steps.default.yamlare touched.Why a separate, additive PR
The shipped
jinli_lme.yamlonly registeredupdate_index/auto_memory/version/search, leaving benchmark authors to either re-derive the wiring or write a duplicate config. Adding a dedicated job matches #326's intent, is one self-contained stanza, and is documented in the newdocs/en|lme_benchmark.md/zhpages (PR #332) as the canonical harness entry point.The job is opt-in: existing jobs in
jinli_lme.yamlkeep their behavior, andreme start config=jinli_lme.yamldoes not auto-register any new dispatch. Only an explicitreme app lme_one_question ...invocation reaches the new path.Verification
python -c 'import yaml; yaml.safe_load(open("reme/config/jinli_lme.yaml"))'→ loads withjobs.lme_one_questionpresent, steps[context_answer_step, answer_judge_step], all six parameters required.pre-commit run --files reme/config/jinli_lme.yaml→ all hooks green (yaml check + pyroma only; Python-side hooks skipped as expected for a YAML-only diff).