Skip to content

feat(search): add containment-style tags filter for multi-user scoping (#368) - #402

Closed
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:feat/issue-368-tag-filter-search
Closed

feat(search): add containment-style tags filter for multi-user scoping (#368)#402
RerankerGuo wants to merge 1 commit into
agentscope-ai:mainfrom
RerankerGuo:feat/issue-368-tag-filter-search

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

🎯 背景 (Background)

Closes #368

问题摘要:Issue 作者报告 v0.4.x 版本不再支持显式的用户维度检索。Maintainer 在 评论 中明确给出了推荐方案:在 Markdown frontmatter 中写入

---
tags:
  - user:alice
---

并在 search 上扩展支持 containment-style tag filter。但当时 search 侧只支持 metadata 的精确相等匹配,不支持对 list 类型的 tags 做包含匹配,因此该方案实际上不可落地。本 PR 补齐这一能力,并把 tags 作为顶级 context 参数暴露,与 start_date/end_date 对齐。


📝 改动内容 (What & Why)

做了什么

  • 新增 containment 语义的 tags filter (LocalFileStore._tag_contains):chunk 必须包含全部期望 tags(AND 语义),同时兼容实际 metadata 为 list / scalar / None 三种情况
  • 在 _matches_search_filter 中保留 tags 作为保留 key,避免走默认的 exact equality,保证其他 metadata key(如 conversation_date)行为不变;同时把末尾两个 early-return 合并为 passes_tags & passes_metadata,把 return 数量控制在 pylint 阈值内
  • SearchStep 顶级 tags 参数 (search.py):与 start_date/end_date 同样 promote 成 search_filter 子项;支持 list[str] 和 CSV string(自动 trim)
  • SearchV2Step 顶级 tags 参数 (search_v2.py):与 SearchStep 保持完全一致
  • 4 个新增单元测试 (test_search_step.py):
    1. filter 语义覆盖用户维度过滤、多 tag AND、无 tags chunk、空 filter 等场景
    2. list 和 CSV 两种输入形式均被正确 promote
    3. 显式 search_filter 优先于 context 顶层
    4. SearchV2Step promotion 通路

为什么这么做

  • AND containment 符合 frontmatter 设计意图:chunk 可以同时带有 user:alice + project:foo,调用方只需要传其中一个就做用户维度隔离,需要更严格就同时传两个
  • 保留 key(而非把整个 metadata 都改成包含语义)维持向后兼容:conversation_date 等标量字段继续用 exact equality
  • 与 start_date/end_date 同用 "promote to search_filter" 风格,调用侧心智模型一致,MCP 客户端直接传 tags 参数即可

明确做什么(边界)

  • 不修改 include_frontmatter_in_metadata 默认值(仍为 false):开启它会放大内存占用,属于部署配置决策,调用方按 maintainer 的方案需要自行打开,或单独开 PR 补配置文档
  • 不修改 BM25 / 向量索引:tags 保持在 post-hoc 过滤层,和 path、date、metadata filters 保持一致,避免引入索引格式迁移
  • 不新增 vector search 端 filter 接口:FaissLocalFileStore 继承自 LocalFileStore,使用同一份 _matches_search_filter 路径,自动获得 tags filter 能力

✅ 验证方式 (Verification)

本地测试

  • 新增 4 个测试通过:pytest tests/unit/test_search_step.py -k "tag_contains or tags_context or tags_in_search" — 4/4 PASS
  • 完整 search 套件回归:pytest tests/unit/test_search_step.py -v — 36/36 PASS(无回归)
  • Pre-commit 全部通过:check-ast、black、flake8、pylint、pyroma、add-trailing-comma、trim-whitespace、detect-private-key — 全部 PASS
  • 无 lint / type 诊断报错

手动(集成侧)验证步骤

对部署方而言,落地用户维度隔离需要同时做两件事:

  1. Markdown chunker 打开 frontmatter 到 metadata(config YAML 中):

    file_chunker:
      markdown:
        include_frontmatter_in_metadata: true
        include_frontmatter_keys_in_metadata: ["tags"]
  2. 搜索传 tags 参数

    reme search query="..." tags="user:alice"

或者等价 MCP 调用:tools/call with name=search, params={query, limit, tags:["user:alice"]}。


🌊 影响范围 (Impact)

  • Breaking Change? → ⚪ 没有
    • 新增可选参数;默认行为(不传 tags)完全等价
    • 其他 metadata filter 仍保持 exact equality,不受 tags 保留 key 影响
  • 受影响模块:local_file_store.py、search.py、search_v2.py
    • FaissLocalFileStore 通过继承自动获得相同能力
  • 性能影响:🟢 可忽略(单一集合 issubset 判断,且 chunk candidate 已经是向量/BM25 召回后的集合)
  • 文档更新:frontmatter tags 用法和 include_frontmatter_in_metadata 配置建议建议在下一个独立文档 PR 补入 MCP / 集成指南

📋 Checklist

  • 我的代码遵循项目的代码风格(运行过 lint / pre-commit 全套)
  • 我加了测试证明我的改动有效(filter 语义 + 两个 search step promotion 通路 + precedence 规则)
  • 新增/修改的测试本地通过,且原搜索套件无回归(36/36 PASS)
  • 本 PR 只解决 1 个 issue(新版本不支持根据用户查询了吗 #368),没有夹带无关改动

@CLAassistant

CLAassistant commented Jul 29, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

#368)

Closes #368

- Add containment-style tags filter to LocalFileStore._matches_search_filter with AND semantics (chunk must contain every expected tag). Matches the design note in the issue: frontmatter tags: [user:alice] plus the markdown chunker include_frontmatter_in_metadata=true lets multi-agent deployments scope recalls to a single user without per-user workspaces.
- Promote tags context parameter in SearchStep and SearchV2Step to search_filter[tags], mirroring the existing start_date/end_date promotion. Accept both list[str] and comma-separated strings (trimmed). Explicit search_filter.tags takes precedence.
- Add _tag_contains helper on LocalFileStore (covers list/set/tuple, scalar, and None inputs) and move the final two early-returns of _matches_search_filter to a combined boolean so pylint keeps the function's return-count under the limit.
- Unit tests: containment semantics at the filter layer, list+CSV promotion in both SearchStep and SearchV2Step, precedence over top-level context.

- The v0.4.x user-facing change removed an explicit user-scoping parameter. The documented workaround (separate workspaces per user) is heavyweight; the maintainer suggestion (store user:<name> in frontmatter tags) needed the corresponding search-side filter to be actionable in practice.
- Using containment (AND) rather than exact equality keeps the frontmatter format natural: mix of user:<name>, project:<proj>, domain tags still works without calling multiple filters.
- Using a reserved top-level tags key keeps the behavior explicit and the metadata semantics stable (metadata keys still use exact equality, which is what callers need for conversation_date style filters).

- Does NOT flip include_frontmatter_in_metadata default to true — that's a deploy-time config decision and changes memory usage. Docs update to note the knob is tracked separately.
- No vector or keyword index changes; filtering stays post-hoc consistent with date/path/metadata filters today.

- [x] New unit tests (4) for filter semantics, list and CSV promotion, precedence, SearchV2Step — all PASS
- [x] Existing search step test suite — 36/36 PASS (no regressions)
- [x] Pre-commit: check-ast, black, flake8, pylint, pyroma, add-trailing-comma, whitespace, private-key — all PASS
@RerankerGuo
RerankerGuo force-pushed the feat/issue-368-tag-filter-search branch from c84bdf1 to c210995 Compare August 6, 2026 08:16
@RerankerGuo RerankerGuo closed this by deleting the head repository Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

新版本不支持根据用户查询了吗

2 participants