fix(converters): preserve literal NA strings in XLSXToDocument (#12945) by HarshitR2004 · Pull Request #13044 · deepset-ai/haystack

HarshitR2004 · 2026-09-30T17:50:45Z

Related Issues

Fixes XLSXToDocument drops literal NA and N/A cell values #12945.

Proposed Changes

XLSXToDocument preserves literal NA strings when callers have not configured NA parsing, while retaining pandas' existing behavior for explicit NA settings.

Apply keep_default_na=False and na_values=[""] only when neither option is present in read_excel_kwargs. This keeps literal "NA", "N/A", "NULL", and "None" while still recognizing empty cells as missing.
When either key is supplied, pass the caller's NA settings through unchanged. Custom na_values alone supplement pandas' built-in defaults; explicit keep_default_na=False keeps its existing empty-cell and numeric-formatting behavior.
Cover lists, empty lists, tuples, sets, strings, column-specific dictionaries, and None in CSV and Markdown. Test explicit overrides, na_filter=False, unrelated read options, numeric columns with gaps, and unchanged caller configuration.
Keep the shared CSV/Markdown assertion, clarify the docstring and current documentation, and document the upgrade path. Restore the archived 3.3 page to match upstream.

How did you test it?

Validation used a workspace-local Hatch environment with Python 3.12.10, pandas 3.0.6, openpyxl 3.1.5, and tabulate 0.10.0.

Before the compatibility fix, the expanded compatibility tests had 22 failures and 8 passes. Those cases now pass.
hatch -e review run pytest test/components/converters/test_xlsx_to_document.py -q --tb=short: 66 passed, 1 failed, including all 38 NA-parsing cases passing.
The remaining failure is the existing test_run_markdown_escapes_cell_content[windows-line-break] case: two spaces are produced instead of one. It was reproduced using the upstream converter from current base c41c45de0648e279efbd6d988eae638d1b85f7df in the same environment.
Ruff lint/format checks and repository pre-commit checks passed for the changed files.
mypy --follow-imports=silent haystack/components/converters/xlsx.py test/components/converters/test_xlsx_to_document.py: passed.
Verified that the archived version 3.3 documentation page has no diff against the upstream base.

Notes for the reviewer

The review follow-up was implemented with AI assistance. The local results above distinguish the passing NA-parsing coverage from the reproduced upstream line-ending failure.

Checklist

I have read the contributors guidelines and code of conduct.
I have added unit tests and updated the docstrings.
The PR title follows the conventional commit format.
I have documented the changed behavior.
I have added a release note, including an upgrade section.
I have run the relevant local checks and reported their results above.

…et-ai#12945)

vercel · 2026-09-30T17:50:52Z

@HarshitR2004 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

CLAassistant · 2026-09-30T17:50:55Z

All committers have signed the CLA.

github-actions · 2026-09-30T17:51:00Z

Hi @HarshitR2004, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12945 is already being addressed by open pull request(s) #12951. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

bogdankostic

Thanks for working on this @HarshitR2004!

Defaulting to keep_default_na=False fixes #12945, but it breaks empty-cell handling for users who pass their own na_values. On main, these are added to pandas' default list, which includes "". With this PR, they replace the [""] default, so empty cells are no longer treated as missing:

# sheet with an empty cell in column B and "foo" in column A
converter = XLSXToDocument(
    read_excel_kwargs={"na_values": ["foo"]},
    table_format_kwargs={"na_rep": "<missing>"},
)
# main:
# 2,x,<missing>
# 3,<missing>,y
# this PR:
# 2,x,
# 3,<missing>,y

Same for missingval with table_format="markdown". For lists, adding "" to the user-provided na_values fixes this.

Apart from this, let's add the upgrade note and the test for passing na_values.

bogdankostic · 2026-10-20T09:51:23Z

+            By default, `keep_default_na` is set to False and `na_values` to `[""]` so literal text
+            values like "NA" and "N/A" are preserved while empty cells remain empty.


Let's also mention what happens when na_values is passed. Right now this reads as if it's added to the [""] default, but it replaces it (see my comment on _extract_tables).

bogdankostic · 2026-10-20T09:51:23Z

@@ -0,0 +1,7 @@
+---
+fixes:


This changes the output for everyone using the default settings, e.g. cells containing "NULL" or "None" now end up in the content instead of being emptied. Let's add an upgrade section explaining how to get the old behavior back.

bogdankostic · 2026-10-20T09:51:23Z

+        if table_format == "csv":
+            rows = [row[1:] for row in list(csv.reader(io.StringIO(content)))[1:]]
+            assert rows == [
+                ["Code", "Status", "Blank"],
+                ["NA", "OK", ""],
+                ["N/A", "pending", ""],
+                ["North America", "active", ""],
+            ]
+        else:
+            rows = [[cell.strip() for cell in row.split("|")[2:-1]] for row in content.splitlines()[2:]]
+            assert rows == [
+                ["Code", "Status", "Blank"],
+                ["NA", "OK", ""],
+                ["N/A", "pending", ""],
+                ["North America", "active", ""],
+            ]


nit: the expected rows are the same for both formats, so let's assert once after parsing:

Suggested change

if table_format == "csv":

rows = [row[1:] for row in list(csv.reader(io.StringIO(content)))[1:]]

assert rows == [

["Code", "Status", "Blank"],

["NA", "OK", ""],

["N/A", "pending", ""],

["North America", "active", ""],

]

else:

rows = [[cell.strip() for cell in row.split("|")[2:-1]] for row in content.splitlines()[2:]]

assert rows == [

["Code", "Status", "Blank"],

["NA", "OK", ""],

["N/A", "pending", ""],

["North America", "active", ""],

]

if table_format == "csv":

rows = [row[1:] for row in list(csv.reader(io.StringIO(content)))[1:]]

else:

rows = [[cell.strip() for cell in row.split("|")[2:-1]] for row in content.splitlines()[2:]]

assert rows == [

["Code", "Status", "Blank"],

["NA", "OK", ""],

["N/A", "pending", ""],

["North America", "active", ""],

]

bogdankostic · 2026-10-20T09:51:24Z

+                ["North America", "active", ""],
+            ]
+
+    def test_run_keep_default_na_override(self, tmp_path: Path) -> None:


Let's also add a test that passes na_values via read_excel_kwargs and checks that empty cells still get na_rep/missingval. That covers the case from my comment on _extract_tables.

HarshitR2004 · 2026-10-20T14:08:25Z

@bogdankostic Thanks for the review! I’ve fixed empty-cell handling with custom na_values, added regression tests, updated the docs and upgrade note, and simplified the assertion as suggested. All new tests and quality checks pass. One existing line-ending test still fails on the unchanged code, which I’ve noted in the PR.

davidsbatista · 2026-10-22T10:14:55Z

Thanks for the updates @HarshitR2004!

The list case works now, but other forms of na_values still break empty cells, and the docs need to be moved to the right version

davidsbatista · 2026-10-22T10:16:46Z

        """
        file_bytes = io.BytesIO(bytestream.data)
        resolved_read_excel_kwargs = {
+            "keep_default_na": False,


Two behaviour changes the upgrade note should mention (the suggestion above avoids the second one):

read_excel_kwargs={"na_values": [...]} used to add to pandas' default NA strings. Now it replaces them, so NA/NULL text stays in the output.

read_excel_kwargs={"keep_default_na": False}: we now also add na_values=[""], so empty cells become NaN and numeric columns with gaps turn into floats.

davidsbatista · 2026-10-22T10:17:48Z

            "engine": "openpyxl",  # Use openpyxl as the engine to read the Excel file
        }
+        na_values = resolved_read_excel_kwargs["na_values"]
+        if isinstance(na_values, list) and "" not in na_values:


"" only gets added when na_values is a list. If someone passes a tuple, set, string, dict or None, empty cells come through as "" strings and na_rep/missingval won't apply to them.

XLSXToDocument(read_excel_kwargs={"na_values": ("foo",)}) # blank cells no longer rendered as missing

A simpler way to handle every case: only set our defaults when the user passes neither key, and otherwise leave their kwargs untouched.

if "keep_default_na" not in self.read_excel_kwargs and "na_values" not in self.read_excel_kwargs: resolved_read_excel_kwargs.update(keep_default_na=False, na_values=[""])

github-actions · 2026-10-22T10:18:11Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/converters
xlsx.py
Project Total

_{This report was generated by python-coverage-comment-action}

davidsbatista · 2026-10-22T10:18:46Z

+            ["North America", "active", ""],
+        ]
+
+    @pytest.mark.parametrize("table_format", ["csv", "markdown"])


Could you parametrize this test over a tuple, a dict and None for na_values as well? Those are the cases that break right now.

davidsbatista · 2026-10-22T10:19:09Z


 To see the additional parameters that you can specify with the component initialization, check out the [API Reference](/reference/converters-api#xlsxtodocument).

+### Missing values


Please revert this file. The change ships in 3.5, so 3.3 users would read about behaviour they don't have. Updating docs-website/docs/ is enough.

HaystackBot · 2026-10-22T10:30:27Z

Hi @HarshitR2004, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

HaystackBot · 2026-10-22T10:53:36Z

Hi @HarshitR2004, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the license/cla check, and this PR will automatically be marked ready for review.

HaystackBot · 2026-10-22T11:59:33Z

Thanks for signing the CLA, @HarshitR2004! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

HarshitR2004 · 2026-10-22T12:27:34Z

@davidsbatista Thanks for the feedback! I’ve applied the suggested conditional defaults, preserving pandas behavior whenever either NA option is supplied. Added tests for all supported input forms, explicit overrides, and numeric formatting; updated the docstring and upgrade note and reverted the 3.3 docs change.

vercel · 2026-10-22T13:20:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
haystack-docs	Ready	Preview	Oct 22, 2026 1:24pm UTC

davidsbatista

thanks for the contribution!

davidsbatista · 2026-10-22T14:35:11Z

@bogdankostic have a look when you have the time

fix(converters): preserve literal NA strings in XLSXToDocument (deeps…

e73435f

…et-ai#12945)

HarshitR2004 requested a review from a team as a code owner September 30, 2026 17:50

HarshitR2004 requested review from bogdankostic and removed request for a team September 30, 2026 17:50

github-actions Bot added the topic:tests label Sep 30, 2026

julian-risch mentioned this pull request Oct 1, 2026

fix(XLSXToDocument): keep cells whose text is a default NA string #12951

Closed

Merge branch 'main' into fix-xlsx-drops-literal-na-values

4aa9424

github-actions Bot added the P3 Low priority, leave it in the backlog label Oct 8, 2026

github-actions Bot mentioned this pull request Oct 19, 2026

Preserve literal NA and N/A values in XLSX converter #13335

Closed

bogdankostic requested changes Oct 20, 2026

View reviewed changes

HarshitR2004 added 2 commits October 20, 2026 19:29

fix(converters): retain empty cells with custom XLSX NA values

b75c940

Merge branch 'main' into fix-xlsx-drops-literal-na-values

bd8af30

github-actions Bot mentioned this pull request Oct 22, 2026

fix: preserve literal NA strings in XLSXToDocument #13389

Closed

5 tasks done

Merge branch 'main' into fix-xlsx-drops-literal-na-values

16349a4

github-actions Bot added the type:documentation Improvements on the docs label Oct 22, 2026

davidsbatista reviewed Oct 22, 2026

View reviewed changes

HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026

HaystackBot marked this pull request as draft October 22, 2026 10:30

davidsbatista removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026

davidsbatista marked this pull request as ready for review October 22, 2026 11:03

Merge branch 'main' into fix-xlsx-drops-literal-na-values

a43932d

HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026

HaystackBot marked this pull request as draft October 22, 2026 11:29

HaystackBot marked this pull request as ready for review October 22, 2026 11:59

HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Oct 22, 2026

fix(converters): preserve explicit XLSX NA parsing settings

4cf7876

davidsbatista reviewed Oct 22, 2026

View reviewed changes

Comment thread docs-website/docs/pipeline-components/converters/xlsxtodocument.mdx Outdated

Apply suggestion from @davidsbatista

7ad32c7

vercel Bot deployed to Preview October 22, 2026 13:24

View deployment

davidsbatista approved these changes Oct 22, 2026

View reviewed changes

davidsbatista requested a review from bogdankostic October 22, 2026 13:32

github-actions Bot mentioned this pull request Oct 28, 2026

fix: keep literal NA and N/A cell values in XLSXToDocument #12945 #13462

Open

		By default, `keep_default_na` is set to False and `na_values` to `[""]` so literal text
		values like "NA" and "N/A" are preserved while empty cells remain empty.


		To see the additional parameters that you can specify with the component initialization, check out the [API Reference](/reference/converters-api#xlsxtodocument).

		### Missing values

		@@ -0,0 +1,7 @@
		---
		fixes:

Conversation

HarshitR2004 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues

Proposed Changes

How did you test it?

Notes for the reviewer

Checklist

Uh oh!

vercel Bot commented Sep 30, 2026

Uh oh!

CLAassistant commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

bogdankostic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HarshitR2004 commented Oct 20, 2026

Uh oh!

davidsbatista commented Oct 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HaystackBot commented Oct 22, 2026

Uh oh!

HaystackBot commented Oct 22, 2026

Uh oh!

HaystackBot commented Oct 22, 2026

Uh oh!

HarshitR2004 commented Oct 22, 2026

Uh oh!

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidsbatista left a comment

Choose a reason for hiding this comment

Uh oh!

davidsbatista commented Oct 22, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HarshitR2004 commented Sep 30, 2026 •

edited

Loading

CLAassistant commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading