[enhancement] Build the reproducible evaluation and benchmark suite

[aotenjou]

Phase 4 of Parent epic: #863

Summary

Provide an offline, reproducible evaluation pipeline for evidence retrieval, answer quality, explainability, latency, and resource cost. Run the required comparison of all retrieval strategies on HotpotQA and 2WikiMultiHopQA under identical data, result, and budget protocols.

Motivation

Retrieval quality cannot be judged from a few examples or from answer accuracy alone. The project needs a fixed evaluation protocol that makes evidence coverage, graph-path quality, latency, and token/candidate costs comparable and allows results to be reproduced from versioned inputs.

Scope

• Implement dataset adapters for the pinned HotpotQA distractor and 2WikiMultiHopQA splits.
• Parse supporting facts and answer annotations into evaluation targets without committing raw datasets to the repository.
• Implement metrics for Evidence Recall@K, MRR, NDCG, extractive Answer EM/F1 or Accuracy, relation-path hit rate, P50/P95 latency, candidate counts, graph-edge scans, and token cost.
• Build a batch runner that records dataset, graph/index, preprocessing, policy, configuration, estimator, code revision, and random-seed versions alongside every result.
• Run six required baselines/methods: BM25-only, Vector-only, Graph-only, Fixed Hybrid, Adaptive Hybrid, and equal-budget Vector Overfetch.
• Run multiple hard-budget levels and query-category ablations, including no-anchor, high-degree, lexical, semantic, and multi-hop cases where applicable.
• Re-run the primary experiments to check deterministic outputs and investigate variance or failures.
• Generate machine-readable JSONL/CSV results and a human-readable comparison report with negative results, failure cases, applicability boundaries, and performance bottlenecks.

Suggested implementation locations

• tools/graphrag/eval/ - dataset adapters, target extraction, and metric implementations.
• tools/graphrag/benchmark/ - batch runner, configurations, result schemas, and report generation.
• tools/graphrag/manifests/ - pinned dataset and preprocessing manifests with checksums.
• geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/ - Java-side trace and cost contract tests.
• docs/ or the repository-approved documentation location - evaluation protocol and reproduction guide.

Testing and reproducibility requirements

• Unit tests for answer normalization, supporting-fact matching, ranking metrics, path matching, percentile latency, and token estimation.
• Golden-result tests for a small fixed query set.
• Smoke tests for the full batch command and schema validation of generated JSONL.
• Pin random seeds, policy versions, estimator versions, and dataset checksums in benchmark configs.

Dependencies and follow-up

• Any benchmark defect that changes a primary metric must be fixed and the affected runs repeated.