feat(sandbox): add isolated two-phase evaluation and Meta-ReMe workspace tooling - #434
Closed
xyf2020 wants to merge 44 commits into
Closed
feat(sandbox): add isolated two-phase evaluation and Meta-ReMe workspace tooling#434xyf2020 wants to merge 44 commits into
xyf2020 wants to merge 44 commits into
Conversation
- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all - record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs - update agentic_answer step and utils exports; add unit tests for job counting and counter utils
…ls in benchmarks - Extract _counter_key from BaseJob._record_call for reusable counter lookup - Add reme.utils.evaluation_interface.check_job_count read-only helper - Track and report average search calls per query in beam and longmemeval benchmarks
- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出 - 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring - black 格式化
This reverts commit 85bf320.
This reverts commit 6722c24.
…benchmarks - Add InjectionConfig passthrough in AsAgentWrapper.reply() - Disable inject_runtime_state in BaseAgenticAnswerStep to avoid wall-clock time conflicting with benchmark query_time anchors - Disable inject_runtime_state in beam/lme llm_judge calls
- Add version_tuple utility for semantic version comparison - AsAgentWrapper: version-aware InjectionConfig, max_iters doubling, and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5 - Default inject_runtime_state=False in wrapper to avoid benchmark time-anchor conflicts; remove per-callsite injection_config overrides - longmemeval run.py: support question_ids filter in dataset config - Fix unused import in test_evaluation_interface; format fixes
fix(evaluation): simplify job counters and make token accounting atomic
…operations Enhance the Docker benchmark sandbox so one container can process multiple cases sequentially and export only analysis-relevant artifacts: - Add reset_case() to discard the active case while keeping the installed candidate, venv, and worker for the next case. - Add export(profile="analysis") (default) and export_full() to choose between a selective analysis archive and the complete disposable case tree; analysis export omits rebuildable indexes, caches, temp files, and source resources. - Introduce runtime-layout.json written by the worker with validated, non-secret, case-relative paths so the host can build safe tar excludes. - Serialize job, export, and reset operations with a per-case lock. - Pin a sandbox-runtime extra in pyproject.toml that omits cookbook-only tushare, dingtalk-stream, pypdf, and polars, and install it in Dockerfile.base. - Set TMPDIR to a case-local directory so subprocess temp files stay inside the disposable case root. - Add unit tests for the new behaviors and a real-Docker LongMemEval integration smoke test.
Add commit_memory_history() so the host controls when daily-memory changes are checkpointed into the runtime workspace's local Git repo. Session ingestion no longer commits implicitly; the host groups one or many sessions into an explicit boundary with a chosen message. - Initialize a Git repo in the runtime workspace on case setup - Validate the configured daily_dir and stage only that path - Allow empty commits so boundaries are always recorded - Reset removes the old Git repo and reinitializes for the next case - Export retains .git for offline history inspection
Add the meta-reme subsystem that generates minimal, benchmark-specific ReMe source bundles from an allowlist. The bundle_builder.py reads build_bundle.yaml to produce three targets (default, lme, beam), each validated by import/step-registration checks before being staged. Also include the design document describing the search-driven harness optimization loop and a sandbox integration test exercising the first LongMemEval and BEAM 100K cases against generated bundles.
…pace init - Add meta-reme/run.py entry point loading a single --config YAML and preparing the workspace via the data_preparation module. - Add meta-reme/data_preparation/ (basic, lme, beam) normalizing BEAM and LongMemEval training cases into the shared CaseSpec/SessionSpec/QuerySpec contracts with reproducibility manifests and read-only case files. - Add per-benchmark config_meta_reme.yaml and run_meta.py launchers for beam and longmemeval that forward to the unified entry point. - Add tests/unit_sandbox/test_meta_reme_run.py covering case selection, read-only writes, empty-dir init, BEAM variant/time-anchor parsing, and config loading. - Update meta-reme/design.md to reflect the single-config CLI and the renamed data_preparation/ structure.
- Add meta-reme/git_manager.py for immutable baseline Git initialization - Enhance prepare_workspace to build and init benchmark code bundle - Refactor sandbox worker into build/queries modes with shared Application lifecycle and per-query log isolation - Add run_build, run_queries, export_evaluation to DockerReMeSandbox - Add JobRequest and EvaluationQuery data models - Support log_filepath parameter in get_logger for isolated query logs - Add safety validation for case IDs and query artifact IDs - Update tests for build/queries modes, log isolation, and workspace init - Add sandbox README_ZH.md and update design docs
|
sa-buc seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
- Remove retry/attempt model: QueryScheduler failures are now terminal instead of requeued; QueryLease no longer carries an attempt number; SandboxSpec.max_retries defaults to 0 and is deprecated - Flatten workspace layout: drop the attempt-<n>/ directory level so case results live directly under cases/<case_id>/ - Add fail-fast: new --fail-fast CLI flag, validation.fail_fast config key, and ValidationFailFastError that cancels sibling workers and writes a run-level failure.json on the first construction or query error - Clarify module boundaries: meta-reme/run.py prepares workspace and runs initial validation; meta-reme/validation/run.py validates an already prepared workspace - Update design.md, validation README, and both benchmark configs - Update tests for single-execution semantics and add fail-fast tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces the sandbox and workspace foundations required to evaluate and optimize ReMe memory implementations reproducibly.
It adds an isolated Docker-based benchmark environment, a two-phase build/query evaluation workflow, and the initial Meta-ReMe data preparation layer. Together, these changes make it possible to construct memory once, restart the application at a persistence boundary, evaluate multiple queries against the resulting workspace, and export structured artifacts for offline analysis.
What changed
DockerReMeSandboxFactoryandDockerReMeSandboxfor running ReMe candidates inside isolated AgentScope Docker workspaces.run_build()andrun_queries()APIs:Meta-ReMe foundation
The new
meta-rememodules provide strict Pydantic contracts for datasets, validation selections, fingerprints, budgets, harnesses, and persisted run state. YAML-driven entry points now normalize LongMemEval and BEAM training data into a common case/session/query representation, create a locked file-native workspace, and install the normalized search dataset as read-only data.A deterministic bundle builder creates minimal
default,lme, orbeamReMe distributions from an explicit manifest. The generated bundle is validated and initialized as a local Git repository with a stable baseline commit, preparing it for later harness proposal and evaluation stages.Compatibility and safety
Existing single-job sandbox helpers remain available. Candidate snapshots exclude common credentials, caches, datasets, runtime state, and VCS metadata, while archive upload validates paths, duplicates, symbolic links, and special files before extraction.
Docker integration and credential-dependent benchmark tests were not run locally.