Liberal HTML parsing and readability-style text extraction in pure Mojo. No Python dependencies, no FFI.
As of mid-2026 the Mojo ecosystem had no library for parsing HTML or pulling clean article text out of a page. mojo-html fills that gap: a liberal HTML tokenizer and a readability-style extraction layer on top of it. The tokenizer is mojo-feed's battle-tested liberal XML pull parser, adapted for HTML semantics (void elements, raw-text elements, unquoted attributes, the entities that matter for prose). I built it to pull article text out of pages I'm sourcing for research and Chain of Thought show notes: bring a URL's raw bytes in, get a title, metadata, and the main-content text out.
If you know Python's readability (readability-lxml), the extraction workflow maps like this:
Python (readability) |
mojo-html |
|---|---|
doc = Document(html) |
var page = extract(html) |
doc.title() |
page.title |
doc.short_title() / meta |
page.meta_description |
doc.summary() |
main_text_confident(html).text |
One difference worth noting: readability's summary() returns cleaned
HTML, whereas main_text_confident() returns the main-content text (plus
a .confident flag indicating whether extraction was high-confidence).
- Tokenizer (
html.tokenizer.HtmlTokenizer): pull-based event stream (start / end / text / EOF) over an HTML document. Never raises on malformed markup in its default liberal mode.- void elements (
<br>,<img>,<meta>, …) with synthetic end events, and<foo/>self-closing syntax - raw-text elements:
<script>and<style>content is consumed literally (a"</div>"inside a script won't derail the parse);<title>and<textarea>are raw but entity-decoded - unquoted attribute values (
href=foo), valueless attributes, and duplicate attributes (first wins), all lowercased, since HTML names are case-insensitive - the ~40 named entities that actually show up in article prose plus
numeric references, with Windows-1252 remapping of the C1 control
range (
“→ a curly quote) and U+FFFD hardening for invalid codepoints - encoding normalization: UTF-16 BOMs, UTF-8 BOM stripping, encoding
sniffed from
<meta charset>, Latin-1/CP1252 transcoding, and lossy UTF-8 recovery so mojibake never propagates strict=Truemode that reports mismatched/stray/unclosed tags and bad entities with line/column locations (for linting HTML you produce, not for pages you consume)
- void elements (
- Extractor (
html.extract.extract): one pass over the event stream with an open-element stack and nearest-match recovery, so unclosed<p>/<li>and crossed tags degrade gracefully. Returns aPage:title(from<title>, falling back toog:title),meta_description(name=descriptionorog:description),canonical_url(link rel=canonical),lang(<html lang>), block-aware readabletext,headings(h1–h6), andlinks(href+ text). main_text(source): readability-style main-content scoring. Builds a compact node tree in one streaming pass, drops boilerplate subtrees (script/style/nav/aside/footer/form/svg/… and ARIArole=navigation|complementary|dialog|…), then scores paragraph-like nodes (p,td,pre,section,h2–h6, and the highest-leverage rule, any<div>with no block-level children) against class/id keyword tables (positive:content,article,post, …; negative:sidebar,comment,ad-, …), propagates scores up the ancestor chain, and discounts each candidate by its link density. A fast path short-circuits scoring when exactly one<article>/<main>/[role=main]region exists with substantial text.main_text_confident(source): same extraction, plus aconfidentflag that's false when the winning text is under 140 characters or scores under 20, a gate for paywall stubs and other thin pages, so callers can tell "no real content here" apart from a confident extraction instead of trusting a stub.- 63 tests across the tokenizer and extractor, run against five real
fixture pages, plus a fuzz runner that feeds arbitrary/broken HTML
through both
extractandmain_text_confidentlooking for crashes.
- No JavaScript, no CSS, no network. Pages that render client-side yield whatever's in the initial HTML; bring your own bytes.
- No full HTML5 tree construction. The tokenizer emits a flat event stream; the extraction layer recovers structure with a nearest-match stack instead of the spec's insertion modes and adoption-agency algorithm. Right trade for text extraction, wrong one for round-tripping documents.
- Single-winner extraction. The scorer returns one subtree, with no sibling-merge step to stitch in adjacent same-parent blocks. A page whose content is split across sibling containers returns only the densest one.
- Readability's keep-anyway override isn't applied. The upstream
okMaybeItsACandidatelist only guards a per-node removal pass this port omits: boilerplate here is pruned by tag/role drop-set and down-weighted by class/id keywords, not deleted node-by-node, so porting that list to the class-weight step would do nothing.
With pixi:
pixi install
pixi run testOr with uv:
uv venv
uv pip install mojo --index https://whl.modular.com/nightly/simple/ --prerelease allow
.venv/bin/mojo run -I src test/test_extract.mojoRequires a Mojo nightly (>=1.0.0b3).
from html import extract
from html import main_text_confident
def main() raises:
var source = open("page.html", "r").read()
var page = extract(source.copy())
print(page.title)
print(page.meta_description)
for link in page.links:
print(link.href, "->", link.text)
var main = main_text_confident(source^)
if main.confident:
print(main.text)
else:
print("thin page, skipping")pixi run test63 tests across the tokenizer and extraction layer, run against five real
captured pages in test/data/: a Substack article, example.com, a Paul
Graham essay (genuine 1990s-style tag soup), a docs.python.org tutorial
page (high link density), and a danluu.com article (minimal markup, no
class hints). test/fuzz_runner.mojo runs extract and
main_text_confident over 400+ mutated documents with zero crashes:
malformed input either parses liberally or raises a clean error.
Eleven pure-Mojo libraries that mirror familiar Python stdlib and PyPI APIs, filling gaps in the native Mojo ecosystem:
- mojo-xml — general-purpose XML
parsing, an ElementTree-shaped DOM (Python's
xml.etree.ElementTree) - mojo-feed — RSS, Atom, and
JSON Feed parsing (Python's
feedparser) - mojo-captions — SRT and WebVTT subtitle/transcript parsing (no Python stdlib parallel)
- mojo-markdown —
CommonMark markdown parsing (Python's
markdown) - mojo-unicodedata —
Unicode normalization and case folding (Python's
unicodedata) - mojo-diff — text diffing
(Python's
difflib) - mojo-template — a
Jinja-flavored template engine (Python's
jinja2) - mojo-tar — tar archive
reading and writing (Python's
tarfile) - mojo-redis — a Redis
client (Python's
redis-py) - mojo-url — URL parsing
and encoding (Python's
urllib.parse)
Issues and PRs welcome, especially real-world pages that extract wrong
(attach the URL or a snippet) and sibling-merge cases the scorer misses.
Run pixi run test before sending a PR.
Built by Conor Bronsdon — host of Chain of Thought, a podcast about AI agents, infrastructure, and engineering. This library exists to pull clean article text out of pages I'm sourcing for research and show notes. Find me on X or LinkedIn.
This is an independent personal project, not affiliated with, sponsored by, or endorsed by any company. All views expressed are my own.
Licensed under the MIT License.