Skip to content

[enhancement] Implement multi-channel retrieval, graph expansion, and fusion execution #865

Description

@aotenjou

Phase 2 of Parent epic: #863

Summary

Build the retrieval core over the versioned graph and indexes delivered by Phase 1. The core must expose BM25, vector, and entity-anchor channels through one Java API, expand selected anchors by at most two hops under explicit budgets, and return candidates through a common evidence protocol.

This phase also adds deterministic deduplication, fusion, truncation, and sequential/parallel/ cascaded execution. Adaptive policy selection is added in Phase 3, but this phase must provide the executor and plan interfaces that the policy can call later.

Motivation

The existing memory search path is session-oriented and loses channel scores during execution. A shared retrieval core is needed before different modes can be compared fairly or called by a production service.

Scope

  • Implement TextRetriever/Lucene BM25 retrieval with real scores, stable ranks, field mappings, topK, and candidate limits.
  • Implement a replaceable VectorRetriever that accepts precomputed vectors and does not depend on a particular embedding service.
  • Implement EntityAnchorResolver using normalized names, aliases, and entities attached to highly ranked chunks. Graph-only mode must return a structured empty result when no reliable anchor exists.
  • Implement GraphRetriever with one- and two-hop expansion, lazy edge scanning, edge-scan caps, neighbor sampling, relation-type priority, deterministic sampling, and diffusion stopping rules.
  • Convert every channel result to the unified Evidence model while preserving raw channel scores, ranks, source/chunk identity, and graph paths.
  • Deduplicate first by stable evidence ID and then by normalized source/chunk identity.
  • Implement weighted reciprocal-rank fusion by default, an optional reranker interface, and final topK/token-budget truncation.
  • Implement sequential, parallel, and cascaded executors with per-stage budget accounting.
  • Support BM25-only, Vector-only, Graph-only, and Fixed Hybrid modes. Expose a validated plan interface for Adaptive Hybrid without embedding routing rules in channel implementations.

Suggested implementation locations

  • geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/ - BM25, vector, and anchor retrievers.
  • geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/graph/ - neighbor provider, frontier, path, sampling, and stopping logic.
  • geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/fusion/ - evidence merger, RRF, reranker SPI, and token truncation.
  • geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/execution/ - stage executor and execution modes.
  • geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/ - channel, graph, fusion, and budget tests using the fixed fixture from Phase 1.

Testing requirements

  • Unit tests for Lucene score preservation, topK, candidate caps, vector ordering, anchor confidence, path scoring, deterministic sampling, and diffusion stopping.
  • Fusion tests for duplicate removal, score/rank preservation, RRF ordering, and token truncation.
  • Concurrency tests for parallel execution and isolation of per-request budget/trace state.
  • End-to-end tests against the Phase 1 in-memory fixture for all four fixed modes and all execution modes.

Dependencies and follow-up

  • Keep channel and executor interfaces independent of Solon so they can also be used by offline benchmark tooling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions