Substack import: posts and subscribers (#24 phase 4)

[nwwatson]

Summary

Phase 4, the final phase of #24: admins can upload Substack's export .zip (Settings → Exports) to import posts and the subscriber list, or upload just the email list .csv. Post content comes from the export itself, so there's no need to scrape the live Substack site.

Stacked on #244 → #243 → #242. This PR targets issue-24-ghost-import; retarget it as the earlier PRs merge. The commit carries Closes #24, so the issue closes when this reaches master.

Changes

Reading the export

• Imports::Substack::ExportReader:
  • Detects a zip vs. a bare CSV by magic bytes.
  • Reads only posts.csv, posts/<post_id>.html and email_list*.csv, matched by path suffix so a wrapping folder still works.
  • Works in memory; nothing is extracted to disk.
  • Zip-bomb limits (entry count, per-entry and total decompressed size) are enforced while decompressing.
  • CSVs: BOM stripped, headers normalized, a required column is checked, and malformed files raise InvalidFile.

Posts

• Imports::Substack::ContentConverter (on the shared converter base):

  • Removes subscribe widgets, paywall markers, SVG icons, and subscribe/share buttons; other buttons become links.
  • Reads embed JSON from data-attrs: YouTube, Vimeo, tweets, embedded posts, Spotify and SoundCloud become links; pullquotes become blockquotes.
  • Downloads the original image encoded in substackcdn.com/image/fetch/<transforms>/<url> through the SSRF-guarded downloader.

• Imports::SubstackImporter:

  Substack	Prose
  post_id (123.slug)	slug
  is_published + post_date	published / scheduled / draft
  Audience everyone / only_free / only_paid & founding	public / members_only / paid_only
  Threads, missing bodies	skipped
  Podcasts	show notes imported, with a warning

Subscribers

• Created with Subscriber.new, not subscribe_or_sign_in!, so no confirmation emails and no subscriber.created webhooks.
• Marked confirmed, with confirmed_at/created_at set to the Substack signup date (growth charts keep their history). email_disabled rows become unsubscribed.
• Existing and invalid emails are skipped and counted.
• Active paid subscribers get a "Substack paid" label, comp/gift ones get "Substack comp" (matched case-insensitively). Their billing stays on Substack, so no membership is created.
• Saved in 500-row transactions for SQLite throughput.

Other

• Import: substack source; ACCEPTED_FILES now allows several extensions per source. The import row shows subscriber counts.
• Adds the csv gem, which Ruby 3.4 no longer ships as a default gem (without it: LoadError at runtime).
• en + es strings

Documentation

• README.md: added a Substack Import feature entry
• CLAUDE.md: import section now covers Substack (subscriber creation rules and why, labels, batch size, reader limits, the csv gem, the converter, the test zip helper)

Testing

• Manual: in Substack go to Settings → Exports → Create new export and download the zip. In Prose, go to Import & Export → Import → Substack and upload it. Check post visibility, subscribers under Subscribers (confirmed, with original dates), and the "Substack paid" label. No emails should be sent. Running it again skips everything.
• 29 new tests: reader (zip/CSV, nested folder, BOM, header normalization, missing column, malformed CSV, corrupt zip, entry/total/count limits), converter (fixture post, CDN originals, button filtering, unsafe/missing data-attrs, Vimeo), importer (post mapping, skips, subscriber confirmation/unsubscribe/dates/labels, existing subscribers untouched, bare CSV, no mail/webhooks/notifications, idempotency), model, job, controller
• Full suite: 1401 runs, 0 failures
• Rubocop: no offenses · Brakeman: no warnings · bundler-audit / importmap audit: clean
• Not yet checked by clicking through in a browser

Out of scope: Ghost members CSV (small follow-up reusing this subscriber logic), Substack open/click stats CSVs, podcast audio, featured images (not included in Substack's export).

Closes #24

🤖 Generated with Claude Code

[nwwatson]

Merged the stack up to date. Added per-item error isolation for Substack in 4a8405a: each post runs inside import_item, and each subscriber row now gets its own savepoint (transaction(requires_new: true)) so an unexpected failure rolls back that row rather than the surrounding batch of 500. Labels are resolved before the savepoint so a rolled-back row can't invalidate a memoized label.