Notebook idea: portable EvalPort format for retrieval-eval query sets

[adhabnr-ux]

Hi — per CONTRIBUTING.md's guidance to open an issue before investing effort into a larger addition, flagging an idea rather than a PR.

I maintain EvalPort (Apache-2.0), an open spec with JSON Schemas for portable LLM/retrieval eval datasets (test suites, test cases, result sets) plus a small Python package (pip install openeval) that validates docs against them and converts to/from a few eval frameworks.

learn/search/semantic-search/semantic-search.ipynb and hello-pinecone-aws.ipynb are great walkthroughs of building an index and running queries, but neither captures the natural next step: a labeled query set (query → expected relevant ids) you could reuse across notebooks or model/embedding changes to check retrieval quality didn't regress. Right now that data would just be an ad hoc Python dict, notebook-specific.

Rough sketch of what such a notebook cell could look like, using the actual EvalPort schema (test_cases[].metadata is free-form, so it's a natural place for relevant-id labels; graders[].type: "code" scores them):

suite = {
    "version": "1.0.0",
    "id": "pinecone_semantic_search_eval",
    "graders": [{"id": "recall", "type": "code",
                 "params": {"language": "python", "source": "score = 1.0 if set(expected) & set(retrieved) else 0.0"}}],
    "test_cases": [
        {"id": "q1", "input": "What is the capital of France?",
         "metadata": {"expected_relevant_ids": ["doc_42", "doc_108"]},
         "graders": ["recall"]}
    ],
}

from openeval.validate import validate_suite
assert validate_suite(suite).valid

results = []
for case in suite["test_cases"]:
    matches = index.query(vector=embed(case["input"]), top_k=5)
    retrieved = [m["id"] for m in matches["matches"]]
    hit = bool(set(case["metadata"]["expected_relevant_ids"]) & set(retrieved))
    results.append({"test_case_id": case["id"], "actual_output": ", ".join(retrieved),
                     "grader_results": [{"grader_id": "recall", "type": "code",
                                         "score": float(hit), "passed": hit}],
                     "passed": hit})

from openeval.convert import create_result_set
result_set = create_result_set(suite, results, run_id="run_001")

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure either way — if this isn't a fit for the repo's scope right now, feel free to close. Happy to draft the notebook myself (under learn/search/ or learn/experimental/) if there's interest.