reindex --embeddings crashes with "too many SQL variables" on large projects (unbatched IN clause in select_by_ids)

[strawberry-code]

Bug

bm reindex --embeddings (and the plain bm reindex, which runs --embeddings internally) crashes on any vault whose project has more entities than SQLite's bound-parameter ceiling, because Repository.select_by_ids() issues a single WHERE id IN (...) query with every id in one call, with no chunking.

On a project with ~34,300 entities, the embedding-sync path collects all pending entity ids and calls find_by_ids(ids) → select_by_ids(session, ids) with the full list in one shot:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables
[SQL: SELECT entity.id, entity.external_id, entity.title, ... FROM entity WHERE entity.id IN (?, ?, ?, ... ~33,814 more ...) AND entity.project_id = ?]

This reproduces reliably: the crash happens immediately (0% into the embedding progress bar), before any embedding work starts, so reindex --embeddings is completely unusable once a project crosses roughly ~999-32,766 params depending on the local SQLite build (macOS system SQLite via Homebrew's bundled Python here caps well under 34k).

Environment

• basic-memory 0.21.1 (Homebrew formula, basicmachines-co/basic-memory)
• Bundled interpreter: CPython 3.14.5 (macOS arm64)
• SQLAlchemy 2.0.44
• macOS 26.5.1 (Sonoma-line, Apple Silicon)
• Project size at time of crash: ~34,300 entities (~1GB sqlite DB, incl. sqlite-vec embeddings)

Root cause

src/basic_memory/repository/repository.py, Repository.select_by_ids():

async def select_by_ids(self, session: AsyncSession, ids: List[int]) -> Sequence[T]:
    """Select multiple entities by IDs using an existing session."""
    query = (
        select(self.Model).where(self.primary_key.in_(ids)).options(*self.get_load_options())
    )
    # Add project filter if applicable
    query = self._add_project_filter(query)

    result = await session.execute(query)
    return result.scalars().all()

Called from sync_service.py's embedding-sync path via find_by_ids(synced_entity_ids), which passes the entire batch of entity ids needing (re-)embedding as one unbounded IN (...) clause.

Suggested fix

Chunk the ids client-side and merge results, same as most ORMs' bulk-fetch helpers do:

async def select_by_ids(self, session: AsyncSession, ids: List[int]) -> Sequence[T]:
    """Select multiple entities by IDs using an existing session."""
    items: List[T] = []
    batch_size = 500  # stay well under SQLite's SQLITE_MAX_VARIABLE_NUMBER
    for i in range(0, len(ids), batch_size):
        batch = ids[i : i + batch_size]
        query = (
            select(self.Model)
            .where(self.primary_key.in_(batch))
            .options(*self.get_load_options())
        )
        query = self._add_project_filter(query)
        result = await session.execute(query)
        items.extend(result.scalars().all())
    return items

I patched this locally (batch_size=500) and re-ran bm reindex --project kv --embeddings on the same 34,300-entity project: completed cleanly, 33900 entities embedded, 31164 skipped, 0 errors. Happy to open a PR with this change (plus a regression test that mocks/asserts chunking above the threshold) if useful: let me know if you'd rather I target a different batch size or use SQLAlchemy's Select.in_() batching helper if one already exists elsewhere in the codebase.

Repro steps

1. Build (or point bm at) a project with >~1000 entities pending embedding sync (crossing the point depends on the local SQLite's compiled SQLITE_MAX_VARIABLE_NUMBER, but reliably reproduces well before 34k).
2. Run bm reindex --project <name> --embeddings (or plain bm reindex).
3. Observe immediate crash at 0% progress with sqlite3.OperationalError: too many SQL variables.