Evaluation & Annotation Tool

[HamedBabaei]

Problem: No proper way to measure experiments.

Task: Build a small web UI for validation.

What to implement

• Show candidate pairs
• Ask human mark: correct / wrong
• Export gold standard
• Compute precision / recall / F1 (if there is any gold)

[cthoyt]

after #123, see https://github.com/cthoyt/sssom-curator

[cthoyt]

@HamedBabaei this is also a problem I have been thinking about recently, see https://cthoyt.com/2026/08/17/mapping-evaluation-implementation.html

[HamedBabaei]

@HamedBabaei this is also a problem I have been thinking about recently, see https://cthoyt.com/2026/08/17/mapping-evaluation-implementation.html

Very nice blog indeed!

The idea of having an annotation tool is also closely related to using LLM-as-a-Judge for assistive tools. I think these should work together as part of the same workflow: the annotation/curation layer can provide the human feedback needed to evaluate, validate, and improve the automated judge.