Skip to content

fix: improve LLMEvaluator input type validation - #13306

Merged
anakin87 merged 3 commits into
deepset-ai:mainfrom
BugBuster18:fix/llm-evaluator-input-type-validation
Oct 23, 2026
Merged

fix: improve LLMEvaluator input type validation#13306
anakin87 merged 3 commits into
deepset-ai:mainfrom
BugBuster18:fix/llm-evaluator-input-type-validation

Conversation

@BugBuster18

@BugBuster18 BugBuster18 commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Fixes input validation in LLMEvaluator so malformed tuples are rejected with ValueError instead of causing an indexing error. It also checks that input types are declared as parameterized list types. Added regression tests for invalid tuple shapes and type declarations, along with a release note.

Tests

  • test/components/evaluators/test_llm_evaluator.py — 28 passed.
  • git diff --check passed.

Fixes #13307

@BugBuster18
BugBuster18 requested a review from a team as a code owner October 17, 2026 10:04
@BugBuster18
BugBuster18 requested review from anakin87 and removed request for a team October 17, 2026 10:04
@vercel

vercel Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

@BugBuster18 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@BugBuster18 BugBuster18 mentioned this pull request Oct 17, 2026
Closed
@HaystackBot

HaystackBot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Hi @BugBuster18, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from anakin87 October 17, 2026 11:25
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 17, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 17, 2026 11:25
@BugBuster18
BugBuster18 marked this pull request as ready for review October 17, 2026 14:27
@HaystackBot
HaystackBot marked this pull request as draft October 17, 2026 14:51
@BugBuster18

BugBuster18 commented Oct 17, 2026

Copy link
Copy Markdown
Contributor Author

I have read the CLA Document and I hereby sign the CLA

@BugBuster18
BugBuster18 marked this pull request as ready for review October 17, 2026 22:00
@HaystackBot
HaystackBot marked this pull request as draft October 17, 2026 22:24
@github-actions github-actions Bot added the P3 Low priority, leave it in the backlog label Oct 19, 2026
@HaystackBot

HaystackBot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Hi @BugBuster18, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the license/cla check, and this PR will automatically be marked ready for review.

@BugBuster18
BugBuster18 marked this pull request as ready for review October 22, 2026 17:12
@HaystackBot
HaystackBot requested a review from anakin87 October 22, 2026 17:16
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026
@HaystackBot

HaystackBot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @BugBuster18! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@HaystackBot
HaystackBot removed the request for review from anakin87 October 22, 2026 17:27
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026
@HaystackBot
HaystackBot marked this pull request as draft October 22, 2026 17:27
@HaystackBot
HaystackBot marked this pull request as ready for review October 23, 2026 10:02
@HaystackBot
HaystackBot requested a review from anakin87 October 23, 2026 10:02
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 23, 2026
@HaystackBot

HaystackBot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @BugBuster18! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@anakin87 anakin87 changed the title Fix LLMEvaluator input type validation fix: improve LLMEvaluator input type validation Oct 23, 2026
anakin87
anakin87 approved these changes Oct 23, 2026
View reviewed changes

@anakin87 anakin87 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

I pushed a few refinements

@anakin87
anakin87 enabled auto-merge (squash) October 23, 2026 10:47
@anakin87
anakin87 merged commit 05dafc4 into deepset-ai:main Oct 23, 2026
24 of 25 checks passed
@github-actions

github-actions Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
  haystack/components/evaluators
  llm_evaluator.py
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anakin87 anakin87 anakin87 approved these changes

Assignees

No one assigned

Labels

P3 Low priority, leave it in the backlog topic:tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

LLMEvaluator accepts malformed input tuples and invalid type declarations

4 participants

@BugBuster18 @CLAassistant @HaystackBot @anakin87