Skip to content

feat(sandbox): add isolated two-phase evaluation and Meta-ReMe workspace tooling - #434

Closed
xyf2020 wants to merge 44 commits into
agentscope-ai:mainfrom
xyf2020:sandbox
Closed

feat(sandbox): add isolated two-phase evaluation and Meta-ReMe workspace tooling#434
xyf2020 wants to merge 44 commits into
agentscope-ai:mainfrom
xyf2020:sandbox

Conversation

@xyf2020

@xyf2020 xyf2020 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR introduces the sandbox and workspace foundations required to evaluate and optimize ReMe memory implementations reproducibly.

It adds an isolated Docker-based benchmark environment, a two-phase build/query evaluation workflow, and the initial Meta-ReMe data preparation layer. Together, these changes make it possible to construct memory once, restart the application at a persistence boundary, evaluate multiple queries against the resulting workspace, and export structured artifacts for offline analysis.

What changed

  • Added DockerReMeSandboxFactory and DockerReMeSandbox for running ReMe candidates inside isolated AgentScope Docker workspaces.
  • Added support for both uploaded source snapshots and candidates preinstalled in Docker images.
  • Introduced batch-oriented run_build() and run_queries() APIs:
    • build jobs share one Application instance;
    • query evaluation starts a fresh Application and reuses it sequentially;
    • individual query failures are recorded without stopping later queries;
    • scores, token deltas, raw results, logs, and errors are written to stable JSON artifacts.
  • Added selective export profiles for analysis, evaluation, full case state, and reusable workspace snapshots.
  • Added workspace upload, container reuse, serialized operations, safe archive handling, and explicit Git checkpoints for memory history.
  • Added configurable exact-path logging so query-specific logs can be captured without unrelated startup or shutdown output.
  • Added Docker build definitions and packaging extras for host-side sandbox control and container runtime dependencies.
  • Added English and Chinese sandbox documentation covering isolation, lifecycle, artifact contracts, and usage examples.

Meta-ReMe foundation

The new meta-reme modules provide strict Pydantic contracts for datasets, validation selections, fingerprints, budgets, harnesses, and persisted run state. YAML-driven entry points now normalize LongMemEval and BEAM training data into a common case/session/query representation, create a locked file-native workspace, and install the normalized search dataset as read-only data.

A deterministic bundle builder creates minimal default, lme, or beam ReMe distributions from an explicit manifest. The generated bundle is validated and initialized as a local Git repository with a stable baseline commit, preparing it for later harness proposal and evaluation stages.

Compatibility and safety

Existing single-job sandbox helpers remain available. Candidate snapshots exclude common credentials, caches, datasets, runtime state, and VCS metadata, while archive upload validates paths, duplicates, symbolic links, and special files before extraction.

Docker integration and credential-dependent benchmark tests were not run locally.

xyf2020 and others added 30 commits July 29, 2026 22:45
- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils
…ls in benchmarks

- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks
- beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出
- 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring
- black 格式化
…benchmarks

- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
  wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls
- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
  and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
  time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes
fix(evaluation): simplify job counters and make token accounting atomic
…operations

Enhance the Docker benchmark sandbox so one container can process
multiple cases sequentially and export only analysis-relevant artifacts:

- Add reset_case() to discard the active case while keeping the
  installed candidate, venv, and worker for the next case.
- Add export(profile="analysis") (default) and export_full() to choose
  between a selective analysis archive and the complete disposable case
  tree; analysis export omits rebuildable indexes, caches, temp files,
  and source resources.
- Introduce runtime-layout.json written by the worker with validated,
  non-secret, case-relative paths so the host can build safe tar
  excludes.
- Serialize job, export, and reset operations with a per-case lock.
- Pin a sandbox-runtime extra in pyproject.toml that omits cookbook-only
  tushare, dingtalk-stream, pypdf, and polars, and install it in
  Dockerfile.base.
- Set TMPDIR to a case-local directory so subprocess temp files stay
  inside the disposable case root.
- Add unit tests for the new behaviors and a real-Docker LongMemEval
  integration smoke test.
Add commit_memory_history() so the host controls when daily-memory
changes are checkpointed into the runtime workspace's local Git repo.
Session ingestion no longer commits implicitly; the host groups one or
many sessions into an explicit boundary with a chosen message.

- Initialize a Git repo in the runtime workspace on case setup
- Validate the configured daily_dir and stage only that path
- Allow empty commits so boundaries are always recorded
- Reset removes the old Git repo and reinitializes for the next case
- Export retains .git for offline history inspection
Add the meta-reme subsystem that generates minimal, benchmark-specific
ReMe source bundles from an allowlist. The bundle_builder.py reads
build_bundle.yaml to produce three targets (default, lme, beam), each
validated by import/step-registration checks before being staged.

Also include the design document describing the search-driven harness
optimization loop and a sandbox integration test exercising the first
LongMemEval and BEAM 100K cases against generated bundles.
xyf2020 added 2 commits August 7, 2026 02:06
…pace init

- Add meta-reme/run.py entry point loading a single --config YAML and
  preparing the workspace via the data_preparation module.
- Add meta-reme/data_preparation/ (basic, lme, beam) normalizing BEAM and
  LongMemEval training cases into the shared CaseSpec/SessionSpec/QuerySpec
  contracts with reproducibility manifests and read-only case files.
- Add per-benchmark config_meta_reme.yaml and run_meta.py launchers for
  beam and longmemeval that forward to the unified entry point.
- Add tests/unit_sandbox/test_meta_reme_run.py covering case selection,
  read-only writes, empty-dir init, BEAM variant/time-anchor parsing, and
  config loading.
- Update meta-reme/design.md to reflect the single-config CLI and the
  renamed data_preparation/ structure.
- Add meta-reme/git_manager.py for immutable baseline Git initialization
- Enhance prepare_workspace to build and init benchmark code bundle
- Refactor sandbox worker into build/queries modes with shared Application
  lifecycle and per-query log isolation
- Add run_build, run_queries, export_evaluation to DockerReMeSandbox
- Add JobRequest and EvaluationQuery data models
- Support log_filepath parameter in get_logger for isolated query logs
- Add safety validation for case IDs and query artifact IDs
- Update tests for build/queries modes, log isolation, and workspace init
- Add sandbox README_ZH.md and update design docs
@CLAassistant

CLAassistant commented Aug 7, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ xyf2020
✅ jinliyl
❌ sa-buc


sa-buc seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

xyf2020 added 12 commits August 8, 2026 13:22
- Remove retry/attempt model: QueryScheduler failures are now terminal
  instead of requeued; QueryLease no longer carries an attempt number;
  SandboxSpec.max_retries defaults to 0 and is deprecated
- Flatten workspace layout: drop the attempt-<n>/ directory level so
  case results live directly under cases/<case_id>/
- Add fail-fast: new --fail-fast CLI flag, validation.fail_fast config
  key, and ValidationFailFastError that cancels sibling workers and
  writes a run-level failure.json on the first construction or query
  error
- Clarify module boundaries: meta-reme/run.py prepares workspace and
  runs initial validation; meta-reme/validation/run.py validates an
  already prepared workspace
- Update design.md, validation README, and both benchmark configs
- Update tests for single-execution semantics and add fail-fast tests
@xyf2020 xyf2020 closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants