Text from no-ToUnicode CID fonts with lying glyph names bypasses flag_bad_blocks; garbled text and control bytes ship in markdown

[tryingET]

Text from no-ToUnicode CID fonts with lying glyph names bypasses flag_bad_blocks; garbled text and control bytes ship in markdown

What do you want to change?

When a PDF contains a Type0 / Identity-H subset font without a ToUnicode CMap, whose
embedded glyph names disagree with the drawn outlines (a glyph named exclam actually
draws small-cap A), marker 2.0.0 keeps the embedded text and emits:

1. plausible-looking substitution garbage — 7HAT IS THE PURPOSE for WHAT IS THE PURPOSE
   (glyph IDs one ASCII band down: GID 33 draws A but is named exclam, GID 52 is four
   and draws T, GID 55 is seven and draws W, …);
2. raw C0 control bytes inside the markdown — a comma glyph surfaces as U+000C, a period
   as U+000E, ? as U+001F (invisible in every viewer, but they corrupt downstream
   search/diff/tooling);
3. flattened bullet lists when the bullet glyph of a sibling broken font extracts as a
   stray s.

Real-world example (2008 booklet, 68 pages; one embedded small-caps companion font
damaged ~15 passages):

9OU HAVE A MIND"UT DO YOU KNOW HOW YOUR MIND OPERATES !RE YOU AWARE OF YOUR
PREJUDICES AND PRECONCEPTIONS ...
        → YOU HAVE A MIND. BUT DO YOU KNOW HOW YOUR MIND OPERATES? ARE YOU AWARE ...

introduces us to the lELD OF STUDY 7HAT IS BIOLOGY
        → ... the FIELD OF STUDY: WHAT IS BIOLOGY?      (lELD = fi-ligature glue)

s interpret events from the perspective of multiple views. s find multiple sources ...
        → (nine-item bullet list, flattened into one run-on "s " paragraph)

Why?

• builders/line.py::flag_bad_blocks decides "garbled" through the prose-trained OCR
  error model. Substitution-garbled text is out of its distribution: it is ~90% real
  English words in the correct order, so it does not look like OCR noise and the block is
  never routed to surya.
• The embedded-text path never checks whether the source font has a usable ToUnicode
  mapping, so there is no deterministic signal that the text layer is untrustworthy — even
  though pdfium exposes it, and pdfminer-style extractors surface the same condition as
  (cid:NN) output.
• No control-character sanitization exists on the markdown output path, so U+000C from
  the glyph fallback lands byte-for-byte in the final document.

Observed with marker 2.0.0 at commit 947d768, surya 0.22.1, Linux, LLM features off
(llm_request_count: 0 for every page in the run metadata).

How? (optional)

Tiered proposal:

1. Deterministic per-block signal (main fix): mark characters produced by fonts
   without a usable ToUnicode mapping; in flag_bad_blocks, flag any block containing
   such characters for re-OCR, bypassing the learned model. Cheap, precise, no training
   data needed.
2. Control-char sanitization: strip/replace C0 control bytes (except \t, \n,
   \r) in the markdown renderer. Unconditional bug fix.
3. (Open question, needs isolation) one heading whose text layer was clean
   (Reading Reflectively) came out as Reading Selectively — presumably a re-OCR
   misread replacing correct embedded text. If maintainers are interested we can try to
   minimize a second reproducer once the primary fix lands.

Reproducer (synthetic, no third-party content): the script below builds a one-page
PDF from scratch whose A–Z outlines sit at GIDs 33–58 under StandardEncoding decoy names,
with comma/period/question-mark glyphs named uni000C/uni000E/uni001F, and no
ToUnicode CMap. With poppler 26.08, WHAT IS THE PURPOSE? extracts as
7(!4 )3 4(% 0520/3%\x1f and READ, STUDY, ASK AGAIN. as 2%!$\x0c 345$9\x0c ....

make_lying_font_pdf.py (run: uv run --with fonttools ./make_lying_font_pdf.py --out-dir /tmp/lying-font)

#!/usr/bin/env python3
"""Build a minimal PDF whose embedded font lies about its glyphs.

GID 0=.notdef, 3=space("space"), 12=comma("uni000C"), 14=period("uni000E"),
31=?("uni001F"), 33..58=A..Z outlines named after the StandardEncoding glyph
at that code point (A="exclam", I="parenright", T="four", W="seven", ...).
No ToUnicode CMap. Run pdftotext on the result to observe the damage.
"""
from __future__ import annotations

import argparse
from pathlib import Path
import subprocess
import sys

DECOY_NAMES = {
    33: "exclam", 34: "quotedbl", 35: "numbersign", 36: "dollar",
    37: "percent", 38: "ampersand", 39: "quoteright", 40: "parenleft",
    41: "parenright", 42: "asterisk", 43: "plus", 44: "comma",
    45: "hyphen", 46: "period", 47: "slash", 48: "zero", 49: "one",
    50: "two", 51: "three", 52: "four", 53: "five", 54: "six",
    55: "seven", 56: "eight", 57: "nine", 58: "colon",
}
AGL = {name: chr(code) for code, name in DECOY_NAMES.items()}
AGL.update({"space": " ", "uni000C": "\x0c", "uni000E": "\x0e", "uni001F": "\x1f"})
SPECIAL_CIDS = {3: ("space", "space"), 12: ("uni000C", "comma"),
                14: ("uni000E", "period"), 31: ("uni001F", "question")}
LINES = ["WHAT IS THE PURPOSE?", "READ, STUDY, ASK AGAIN."]
SOURCE_FONTS = [
    "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
    "/usr/share/fonts/Adwaita/AdwaitaSans-Regular.ttf",
    "/usr/share/fonts/liberation/LiberationSans-Regular.ttf",
    "/usr/share/fonts/liberation/LiberationSerif-Regular.ttf",
]


def cid_for(char: str) -> int:
    if char == " ":
        return 3
    if char == ",":
        return 12
    if char == ".":
        return 14
    if char == "?":
        return 31
    if "A" <= char <= "Z":
        return ord(char) - 32
    raise ValueError(f"unsupported character: {char!r}")


def build_font(path: Path) -> None:
    from fontTools.fontBuilder import FontBuilder
    from fontTools.pens.ttGlyphPen import TTGlyphPen
    from fontTools.ttLib import TTFont

    source = next((p for p in SOURCE_FONTS if Path(p).exists()), None)
    if source is None:
        sys.exit("no source outline font found in: " + ", ".join(SOURCE_FONTS))
    src = TTFont(source)
    src_glyphs = src.getGlyphSet()

    layout: dict[int, tuple[str, str]] = {0: (".notdef", ".notdef")}
    for cid, (name, drawn) in SPECIAL_CIDS.items():
        layout[cid] = (name, {"space": "space", "comma": "comma",
                              "period": "period", "question": "question"}[drawn])
    for code, name in DECOY_NAMES.items():
        layout[code] = (name, chr(64 + code - 32))

    order = [layout[gid][0] if gid in layout else f"pad{gid}"
             for gid in range(max(layout) + 1)]
    for gid in range(max(layout) + 1):
        layout.setdefault(gid, (f"pad{gid}", ".notdef"))

    fb = FontBuilder(2048, isTTF=True)
    fb.setupGlyphOrder(order)
    fb.setupCharacterMap({})  # no cmap at all: forces glyph-name fallback
    glyphs, metrics = {}, {}
    for name, drawn in layout.values():
        pen = TTGlyphPen(src_glyphs)
        src_glyphs[drawn].draw(pen)
        glyphs[name] = pen.glyph()
        metrics[name] = (getattr(src_glyphs[drawn], "width", 600), 0)
    fb.setupGlyf(glyphs)
    fb.setupHorizontalMetrics(metrics)
    fb.setupHorizontalHeader(ascent=800, descent=-200)
    fb.setupOS2(sTypoAscender=800, sTypoDescender=-200,
                usWinAscent=800, usWinDescent=200)
    fb.setupNameTable({"familyName": "LieSans", "styleName": "Regular",
                       "uniqueFontIdentifier": "LieSans Regular",
                       "fullName": "LieSans", "psName": "LieSans-Regular",
                       "version": "Version 0.1"})
    fb.setupPost()  # format 2.0: persists the decoy glyph names
    fb.save(str(path))


def build_pdf(path: Path, font_path: Path) -> None:
    font = font_path.read_bytes()
    text_ops = ["BT", "/F1 16 Tf", "72 700 Td"]
    for line in LINES:
        hex_cids = "".join(f"{cid_for(c):04X}" for c in line)
        text_ops += [f"<{hex_cids}> Tj", "0 -28 Td"]
    text_ops.append("ET")
    content = "\n".join(text_ops).encode("ascii")

    objects: list[bytes] = [
        b"<< /Type /Catalog /Pages 2 0 R >>",
        b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
        b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
        b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
        b"<< /Type /Font /Subtype /Type0 /BaseFont /LieSans-Regular "
        b"/Encoding /Identity-H /DescendantFonts [6 0 R] >>",
        b"<< /Length %d >>\nstream\n%s\nendstream" % (len(content), content),
        b"<< /Type /Font /Subtype /CIDFontType2 /BaseFont /LieSans-Regular "
        b"/CIDSystemInfo << /Registry (Adobe) /Ordering (Identity) /Supplement 0 >> "
        b"/FontDescriptor 7 0 R /DW 600 /CIDToGIDMap /Identity >>",
        b"<< /Type /FontDescriptor /FontName /LieSans-Regular /Flags 4 "
        b"/FontBBox [0 -200 1000 900] /ItalicAngle 0 /Ascent 800 /Descent -200 "
        b"/CapHeight 700 /StemV 80 /FontFile2 8 0 R >>",
        b"<< /Length %d /Length1 %d >>\nstream\n%s\nendstream"
        % (len(font), len(font), font),
    ]
    out = bytearray(b"%PDF-1.7\n%\xe2\xe3\xcf\xd3\n")
    offsets = [0]
    for number, body in enumerate(objects, start=1):
        offsets.append(len(out))
        out += b"%d 0 obj\n%s\nendobj\n" % (number, body)
    xref_at = len(out)
    out += b"xref\n0 %d\n" % (len(objects) + 1)
    out += b"0000000000 65535 f \n"
    for offset in offsets[1:]:
        out += b"%010d 00000 n \n" % offset
    out += (b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n"
            % (len(objects) + 1, xref_at))
    path.write_bytes(bytes(out))


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--out-dir", default=".")
    args = ap.parse_args()
    out_dir = Path(args.out_dir).expanduser().absolute()
    out_dir.mkdir(parents=True, exist_ok=True)
    font_path, pdf_path = out_dir / "lying_font.ttf", out_dir / "lying_font.pdf"
    build_font(font_path)
    build_pdf(pdf_path, font_path)
    print(f"wrote {pdf_path} ({pdf_path.stat().st_size} bytes)")
    actual = subprocess.run(["pdftotext", "-layout", str(pdf_path), "-"],
                            capture_output=True, text=True, check=True).stdout
    print(f"pdftotext says:\n  {actual!r}")
    decoy_hits = sum(1 for name in DECOY_NAMES.values() if AGL[name] in actual)
    control_hits = sum(1 for byte in ("\x0c", "\x1f") if byte in actual)
    print(f"decoy glyph chars present: {decoy_hits}/26, control bytes: {control_hits}/2")
    if decoy_hits >= 10 or control_hits:
        print("CONFIRMED: lying glyph names produce garbled/control-char text layer")
    else:
        print("NOT CONFIRMED by this poppler build; still valid for pdfium/marker runs")
        sys.exit(1)


if __name__ == "__main__":
    main()

Suggested regression assertions once fixed:

• a block containing characters from a no-ToUnicode font ends with
  text_extraction_method == "surya";
• the rendered markdown contains no [\x00-\x08\x0b\x0c\x0e-\x1f\x7f].