Skip to content

MarkdownHeaderSplitter keeps the closing # sequence of ATX headers in header metadata #13281

Closed

Description

@TemitopeKadri

Describe the bug

MarkdownHeaderSplitter keeps the optional closing sequence of an ATX header in the header and parent_headers metadata. A header written as ## Install ## is stored as "Install ##", and every child chunk under # Guide # gets parent_headers == ["Guide #"].

In Markdown the closing #s are not part of the heading text (CommonMark, ATX headings: "The optional closing sequence of #s must be preceded by spaces or tabs and may be followed by spaces or tabs only"). Some Markdown writers and doc converters emit this style.

Error message

None. The metadata is silently wrong.

Expected behavior

header == "Install" and parent_headers == ["Guide"]. Text where the #s are part of the heading should stay as it is: ## C# (no whitespace before the #), ## Install #1 (text after the #s) and ## Install \# (escaped). The chunk content should stay byte-exact.

To Reproduce

On current main (aca9fc25d):

from haystack import Document
from haystack.components.preprocessors import MarkdownHeaderSplitter

text = "# Guide #\nIntro.\n## Install ##\nRun pip install.\n"
docs = MarkdownHeaderSplitter().run(documents=[Document(content=text)])["documents"]
print([(d.meta["header"], d.meta["parent_headers"]) for d in docs])
# >> [('Guide #', []), ('Install ##', ['Guide #'])]
# expected [('Guide', []), ('Install', ['Guide'])]

Additional context

Header text is taken from match.group(2).strip() in _split_text_by_markdown_headers, so nothing removes the closing sequence. This affects anything that filters or displays chunks by header/parent_headers, and it means the same section gets different metadata depending on which heading style the source used.

This is separate from the fenced-code-block cases in #12954. I have a small fix with regression tests ready and will open a PR.

System:

  • Haystack version (commit or version number): main @ aca9fc25d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions