SASEvaluator returns negative scores when no cross encoder score in the batch is above 1 #13243

Describe the bug
For Cross Encoders that return raw logits, SASEvaluator only applies a sigmoid if at least one score in the batch is above 1. So the same answer gets a different score depending on the other answers in the batch, and a batch of only wrong answers returns negative scores. This happens with cross-encoder/ms-marco-MiniLM-L-6-v2, the model from the SASEvaluator docstring example.

Error message
No error. The scores are just wrong.

Expected behavior
A pair should get the same score in every batch, and scores should stay between 0 and 1.

Additional context
The check is here:

Wrong answers have negative logits, so a batch without a good answer never passes > 1 and the raw logits are returned.

History: the > 1 check was added in #6877 and kept when the evaluator was rewritten in #7428. Haystack 1.x used the raw model output without it. A reviewer already questioned this line on #6877 (comment) and suggested abs(similarity_scores) > 1, but that would still depend on the batch, for example when all logits are between -1 and 1.

A possible fix is to decide based on the model instead: apply the sigmoid when the Cross Encoder's activation_fn is Identity. I'll open a PR with this fix and tests.

To Reproduce

from haystack.components.evaluators import SASEvaluator

evaluator = SASEvaluator(model="cross-encoder/ms-marco-MiniLM-L-6-v2")
evaluator.warm_up()

print(evaluator.run(ground_truth_answers=["Paris"], predicted_answers=["Lyon"]))
# >> {'score': -9.431, 'individual_scores': [-9.431]}

print(evaluator.run(ground_truth_answers=["Paris", "Paris"], predicted_answers=["Lyon", "Paris"]))
# >> {'score': 0.496, 'individual_scores': [0.0, 0.993]}

"Lyon" vs "Paris" scores -9.431 alone and 0.0 next to a correct answer.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: Windows 11
• Haystack version: main (3.4.0-rc0), also reproduced on haystack-ai 3.3.0