Currently, cache checking during content fetches uses file system heuristics (file existence, If-Modified-Since headers, content-length checks) in mediaDownloader. This works but is fragile for cases where metadata needs to persist across runs or be queried efficiently.
Proposal: Add a SQLite-backed cache database (via better-sqlite3 or similar) to track downloaded asset metadata — URL, etag, last-modified, file hash, local path, fetch timestamp. The cache DB would live under .launchpad/ alongside the existing temp/backup directories.
Benefits:
- Faster cache validation without hitting the filesystem per-file
- Enables accurate cache invalidation by etag/hash rather than just file existence
- Queryable from other tooling (dashboard, diagnostics)
Open questions:
- Should it be per-source or shared across sources?
- Should it survive across clearOldData stages, or be scoped to a run?
- Integration point: pre-download check in fetchSources stage, or inside mediaDownloader transform?
Currently, cache checking during content fetches uses file system heuristics (file existence, If-Modified-Since headers, content-length checks) in mediaDownloader. This works but is fragile for cases where metadata needs to persist across runs or be queried efficiently.
Proposal: Add a SQLite-backed cache database (via better-sqlite3 or similar) to track downloaded asset metadata — URL, etag, last-modified, file hash, local path, fetch timestamp. The cache DB would live under .launchpad/ alongside the existing temp/backup directories.
Benefits:
Open questions: