[Vedant-Agarwal]

Related Issues

• fixes MarkdownHeaderSplitter treats headings inside longer closing fences as headers #12954

Proposed Changes:

_code_block_pattern only accepted a closing fence identical to the opening one, required both fences to start at column 0, and matched nothing when a block was never closed. In all three cases the block wasn't recognised, so # lines inside it were split off as headers.

The pattern now follows the CommonMark fence rules:

• the closing fence uses the same character and is at least as long as the opening one, so can close with ````` ````, but a shorter run or the other character inside the block doesn't close it
• fences can be indented by up to three spaces (four or more is an indented code block, which the splitter doesn't track, same as before)
• a block that is never closed runs to the end of the document
• a backtick fence's info string can't contain a backtick, so a line that starts with inline code (```foo```) doesn't open a block and hide every header after it

Chunk content is unchanged, only which # lines count as headers.

How did you test it?

• Added test_commonmark_fence_rules, 12 parametrised cases: longer backtick and tilde closers (the issue repro), shorter run and other character not closing, a fence line with an info string not closing, unclosed block, indented opener/closer, four spaces not being a fence, inline code not being a fence, tilde info string with backticks, CRLF line endings, and a closer on the last line with no trailing newline.
• 4 of the new cases fail on main and all 12 pass with the fix. The equal-length cases that already worked still pass.
• test_markdown_header_splitter.py: 85 passed, including the tests from fix: drop the ATX closing sequence from MarkdownHeaderSplitter header metadata #13282.
• ruff check and ruff format --check (0.16.0) pass, and mypy reports nothing for the changed file.

Notes for the reviewer

#13282 noted that the fence cases from this issue live in the separate _code_block_pattern, so this PR only touches that pattern and its comment. The comment above the pattern explains each part of the regex.

AI-assisted: implemented with Claude Code. I reviewed the changes and ran the tests above.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@Vedant-Agarwal is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[anakin87]

Mypy error:
test/components/preprocessors/test_markdown_header_splitter.py:674: error: Generator has incompatible item type "str | None"; expected "str" [misc]

[Vedant-Agarwal]

Thanks, fixed in a03c809.

(AI-assisted)

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
markdown_header_splitter.py
Project Total

This report was generated by python-coverage-comment-action

[anakin87]

👍