Skip to content

fix: detect upper-accented+currency mojibake at string start (fixes #222) - #232

Open
gaoflow wants to merge 1 commit into
rspeer:mainfrom
gaoflow:main
Open

fix: detect upper-accented+currency mojibake at string start (fixes #222)#232
gaoflow wants to merge 1 commit into
rspeer:mainfrom
gaoflow:main

Conversation

@gaoflow

@gaoflow gaoflow commented Jun 24, 2026

Copy link
Copy Markdown

The badness heuristic in is_bad() misses UTF-8 mojibake when an upper-accented letter followed by a currency symbol appears at the very beginning of a string. An existing pattern (\s [{upper_accented}] [{currency}]) already catches this when preceded by whitespace, but the start-of-string case was missing.

For example, fix_encoding("Ã¥klagarmyndighets") returns "Ã¥klagarmyndighets" unchanged instead of correctly decoding to "åklagarmyndighets".

Root cause: No BADNESS_RE pattern matches [{upper_accented}][{currency}] at position 0 without a preceding space or lowercase letter.

Fix: Add ^[{upper_accented}][{currency}]\w to detect this sequence at string start. The trailing \w is required so the pattern does not match the isolated 2-character substring that decode_inconsistent_utf8 passes to is_bad(), preventing false positives on ambiguous embedded sequences (like the "Dråber" negative test case from issue #202).

Changes:

  • ftfy/badness.py: Add new BADNESS_RE alternation (4 lines)
  • tests/test-cases/synthetic.json: Add test case for the reported bug

When an upper-accented letter (such as Ã) is followed by a currency
symbol (such as ¥, YEN SIGN) at the very beginning of a string, the
badness heuristic failed to detect it as mojibake.  An existing pattern
(`\s [{upper_accented}] [{currency}]`) already caught this case when
a preceding whitespace was present, but the start-of-string case was
missing.

Add a BADNESS_RE pattern `^[{upper_accented}][{currency}]\w` that
matches this sequence at position 0 only when followed by a word
character.  The trailing `\w` ensures the pattern does not match the
isolated 2-character substring that `decode_inconsistent_utf8` passes
to `is_bad()` during processing of other text, preventing false
positives on ambiguous embedded sequences like "Dråber".

Fixes rspeer#222.
@sebollin

sebollin commented Aug 8, 2026

Copy link
Copy Markdown

I applied this on top of 74dd045 and it does what it says:

  • fix_encoding("Ã¥klagarmyndighets")åklagarmyndighets
  • the whitespace case still works
  • suite unchanged at 351 passed, 10 xfailed

The trailing \w is load-bearing, as the description says. Without it the alternative also matches the two-character fragment that decode_inconsistent_utf8 looks at in isolation — took me a minute to see why it was there.

One thing for whoever reviews this: that rule is the only alternative pairing upper_accented with currency, so (?:^|\s) in place of \s would cover both positions in a single alternative instead of two, if that shape is preferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants