Repository navigation
fix: distinguish empty-string content from None in Document auto-id - #13248
ompatil1906 wants to merge 1 commit into
Conversation
Empty content was normalized with ``or None``, causing Document(content="") and Document(content=None) to share the same ID. This led to spurious DuplicateDocumentError on batch writes and silent ID collisions otherwise. The hash now uses the content value as-is so the two cases are distinct. The stable ID for content=None is unchanged. Fixes deepset-ai#13234
|
@ompatil1906 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
Hi @ompatil1906, thanks for your interest in contributing to Haystack! 🙏 This is an automated message to help us keep the review queue healthy. |
|
Hi @ompatil1906, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
|
Thanks for signing the CLA, @ompatil1906! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
|
Thanks for the PR! We've decided not to change this behavior since it would change the auto-generated IDs of existing documents with empty content. See #13234 for details. Closing. |
Problem
Document._create_idnormalized empty content withtext = self.content or None. ConsequentlyDocument(content="")andDocument(content=None)produced identical auto-generated IDs whenever the rest of their fields matched.This caused:
DuplicateDocumentErrorwhen both were written in the same batch, andEmpty string is a valid
contentvalue (str | None), so the documents are semantically different.Fixes #13234
Solution
Use
text = self.contentdirectly. The stringification then distinguishes""fromNonewhile leaving the stable ID of a completely emptyDocument()unchanged.Verification
test_empty_string_content_has_distinct_id_from_nonethat asserts the two IDs differ and that thecontent=NoneID remains the previously published value.test_init(both passed).Release note added under
releasenotes/notes/.