MRR, MAP and NDCG evaluators give a perfect score to unrelated documents when the comparison value is empty #13258

Describe the bug
DocumentMRREvaluator, DocumentMAPEvaluator and DocumentNDCGEvaluator only skip None comparison values. If the document_comparison_field is an empty string on both a ground truth and a retrieved document (for example meta.file_id="" when ids were never filled in), "" == "" counts as a match and an unrelated document gets a perfect score. DocumentRecallEvaluator already skips empty values, so on the same data Recall reports 0.0 while MRR, MAP and NDCG report 1.0.

Error message
No error. The scores are just wrong.

Expected behavior
An empty comparison value shouldn't match anything, same as in DocumentRecallEvaluator. The example below should score 0.0 for all metrics.

Additional context
The ground truth values are only filtered for None here:

• MRR
• MAP
• NDCG

while Recall filters both (here).

History: #11951 fixed this for Recall and its description mentions that empty strings cause the same false 1.0. It also says MAP, MRR and NDCG already filter None, which is true, but they don't filter "", so the empty-string case was left in these three.

I'll open a PR that skips empty values the same way Recall does, with a test for each evaluator.

To Reproduce

from haystack import Document
from haystack.components.evaluators import (
    DocumentMAPEvaluator,
    DocumentMRREvaluator,
    DocumentNDCGEvaluator,
    DocumentRecallEvaluator,
)

ground_truth = [[Document(content="Strasbourg is the seat of the European Parliament", meta={"file_id": ""})]]
retrieved = [[Document(content="Pasta is cooked in boiling water", meta={"file_id": ""})]]

for evaluator in [
    DocumentRecallEvaluator(document_comparison_field="meta.file_id"),
    DocumentMRREvaluator(document_comparison_field="meta.file_id"),
    DocumentMAPEvaluator(document_comparison_field="meta.file_id"),
    DocumentNDCGEvaluator(document_comparison_field="meta.file_id"),
]:
    result = evaluator.run(ground_truth_documents=ground_truth, retrieved_documents=retrieved)
    print(type(evaluator).__name__, result["score"])
# >> DocumentRecallEvaluator 0.0
# >> DocumentMRREvaluator 1.0
# >> DocumentMAPEvaluator 1.0
# >> DocumentNDCGEvaluator 1.0

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: Windows 11
• Haystack version: main (3.4.0-rc0)