DocumentCleaner(remove_regex=...) strips leading/trailing whitespace of every page, even when the regex matches nothing #13357

Describe the bug

DocumentCleaner._remove_regex calls .strip() on every page after substitution:

cleaned_text = [re.sub(regex, "", text).strip() for text in texts]

So setting remove_regex also strips leading and trailing whitespace (indentation, newlines) from each \f-separated page. This happens whether or not the regex matches anything. It also happens when remove_extra_whitespaces, remove_empty_lines and strip_whitespaces are all False.

Neither the docstring ("Regex to match and replace substrings by """) nor the docs ("any matches will be removed") mention this. remove_substrings and replace_regexes don't strip.

Error message

None. The content is changed silently.

Expected behavior

remove_regex removes only the matched text. Leading and trailing whitespace is controlled by strip_whitespaces, just as it is for replace_regexes.

Additional context

This affects Markdown, code and other indentation-sensitive content, the same use case strip_whitespaces was added for in #10400. It also affects page-aware pipelines that rely on the newlines around \f. DocumentPreprocessor passes remove_regex through, so it behaves the same way. The \f count is preserved, so #8078 is not affected.

The .strip() dates back to the original 2.0 implementation (#5976). I couldn't find any discussion saying it was intended.

To Reproduce

from haystack import Document
from haystack.components.preprocessors import DocumentCleaner

off = dict(remove_empty_lines=False, remove_extra_whitespaces=False, strip_whitespaces=False)

def clean(text, **kw):
    return DocumentCleaner(**off, **kw).run(documents=[Document(content=text)])["documents"][0].content

pattern = r"\[TODO\]"  # does not occur in the inputs below

print(repr(clean("    def f():\n        return 1\n", remove_regex=pattern)))
# 'def f():\n        return 1'                      <- indentation and trailing newline lost

print(repr(clean("\n# Page 1\n\n\f\n## Page 2\n    indented\n", remove_regex=pattern)))
# '# Page 1\x0c## Page 2\n    indented'            <- every page's edges stripped

print(repr(clean("    def f():\n        return 1\n", replace_regexes={pattern: ""})))
# '    def f():\n        return 1\n'                <- same pattern via replace_regexes: unchanged

Possible fixes

I'm happy to open a PR for whichever option you prefer:

1. Remove the .strip() from _remove_regex. Users who relied on it can get the old result back with strip_whitespaces=True. Because cleaned content can change (and so can document IDs), this would ship with an upgrade release note. One example: removing a footer like Page \d$ would leave the newline before it in place. With this change, no existing test in test_document_cleaner.py fails.
2. Docs only. Document that remove_regex strips each page's leading and trailing whitespace, and recommend replace_regexes={pattern: ""} when whitespace must be preserved. This could fit the possible future deprecation of remove_regex in favor of replace_regexes mentioned in feat: add strip_whitespaces and replace_regexes to DocumentCleaner #10400.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: macOS
• Haystack version: main @ 68c782c (3.5.0-rc0)