DocumentRecallEvaluator and DocumentNDCGEvaluator crash on list/dict meta comparison values; MRR and MAP accept them #13200

Describe the bug

With document_comparison_field set to a meta.<key> whose value is a list or dict, DocumentRecallEvaluator and DocumentNDCGEvaluator crash with TypeError: unhashable type, while DocumentMRREvaluator and DocumentMAPEvaluator score the same input correctly.

Recall collects the comparison values into sets (_unique_comparison_values) and NDCG keys a dict by them (_build_relevance_map), so both need hashable values. MRR and MAP compare with in against a list, which works for any value with ==. List-valued metadata is common (source ids, tags, the _split_overlap that DocumentSplitter writes), so the four evaluators accept different inputs for the same document_comparison_field.

Error message

  File "haystack/components/evaluators/document_ndcg.py", line 88, in _build_relevance_map
TypeError: unhashable type: 'list'

(dict values give unhashable type: 'dict'; for Recall the error comes from the set comprehension in _unique_comparison_values.)

Expected behavior

All four evaluators handle the same document_comparison_field values. Here, every one should score 1.0, like MRR and MAP already do.

To Reproduce

from haystack import Document
from haystack.components.evaluators import (
    DocumentMAPEvaluator,
    DocumentMRREvaluator,
    DocumentNDCGEvaluator,
    DocumentRecallEvaluator,
)

ground_truth = [[Document(content="Paris is the capital of France.", meta={"source_ids": ["wiki-1", "wiki-2"]})]]
retrieved = [[Document(content="Paris is the capital of France.", meta={"source_ids": ["wiki-1", "wiki-2"]})]]

for cls in (DocumentMRREvaluator, DocumentMAPEvaluator, DocumentRecallEvaluator, DocumentNDCGEvaluator):
    evaluator = cls(document_comparison_field="meta.source_ids")
    try:
        print(cls.__name__, evaluator.run(ground_truth_documents=ground_truth, retrieved_documents=retrieved)["score"])
    except TypeError as e:
        print(cls.__name__, "TypeError:", e)

Output on main (c5e1335):

DocumentMRREvaluator 1.0
DocumentMAPEvaluator 1.0
DocumentRecallEvaluator TypeError: unhashable type: 'list'
DocumentNDCGEvaluator TypeError: unhashable type: 'list'

Additional context

A small fix is to normalize unhashable comparison values to a hashable key in Recall and NDCG (e.g. recursively converting lists to tuples and dicts to sorted item tuples) before building the set / relevance map, leaving the equality semantics MRR and MAP already use unchanged. I'm happy to open a PR with regression tests for list and dict values in both evaluators if that approach works for you.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: Linux (WSL2)
• Haystack version: main at c5e1335 (3.4.0-rc0)