Skip to content

ews: accelerate FindItem search via grommunio-web's FTS5 index - #341

Open
nourdineb-ops wants to merge 2 commits into
grommunio:masterfrom
nourdineb-ops:ews-fts-search-pr
Open

nourdineb-ops wants to merge 2 commits into
grommunio:masterfrom
nourdineb-ops:ews-fts-search-pr

Conversation

@nourdineb-ops

Copy link
Copy Markdown
Contributor

Outlook's Instant Search box sends its query as the QueryString (AQS) element on FindItem, which gromox never parsed — the element existed only as a schema comment, so the search term was silently dropped and the folder's unfiltered contents were returned and displayed as "results". Subject Contains searches went through gromox's own content-restriction evaluation, a per-row strstr()/strcasestr() scan with no text index, making an "all folders" search from Outlook slow (one FindItem per folder in a burst, each doing a full unindexed scan).

grommunio-web already maintains a per-user SQLite FTS5 index (via the grommunio-index package/timer) for its own search UI. This wires FindItem up to that same index instead of duplicating indexing logic:

  • QueryString is now parsed and used to build a real restriction, scoped to the requested folder. A successful query with zero matches now correctly returns "no results" instead of an unfiltered listing; any failure to open/query the index (no index yet, corrupt, ...) falls back transparently to the previous unrestricted behavior — never a new error.
  • A plain subject Contains (Full string or Prefix) restriction is accelerated the same way. Substring mode is deliberately left alone: FTS5's token/prefix matching can't reproduce arbitrary mid-word substring matches, so accelerating it would silently change semantics. Anything else (AND/OR trees, body/attachment search, other properties) is untouched and keeps using the original restriction.
  • Connections to each user's index.sqlite3 are cached (one per user, for the life of the process) instead of opened per FindItem call; opening fresh on every call in an "all folders" burst caused real, if transient, contention with grommunio-index's periodic regeneration of the same file. The cache is capped (256 entries, LRU-evicted) and retries a failed open after 5 minutes, so a newly-indexed or newly-provisioned mailbox is picked up without a process restart.
  • The index base path is configurable (ews.cfg:ews_fts_index_path, defaults to /var/lib/grommunio-web/sqlite-index); empty disables the acceleration entirely.

Deployed and observed in production for several days before this cleanup pass (config knob, connection-cache hardening, path-safety check) turned it into something PR-ready.

🤖 Generated with Claude Code

nourdineb-ops and others added 2 commits September 8, 2026 08:36
Outlook's Instant Search box sends its query as the QueryString (AQS)
element on FindItem, which gromox never parsed - the element existed only
as a schema comment, so the search term was silently dropped and the
folder's unfiltered contents were returned and displayed as "results".
Subject "Contains" searches went through gromox's own content-restriction
evaluation, a per-row strstr()/strcasestr() scan with no text index,
making an "all folders" search from Outlook slow (one FindItem per folder
in a burst, each doing a full unindexed scan).

grommunio-web already maintains a per-user SQLite FTS5 index (via the
grommunio-index package/timer) for its own search UI. This wires FindItem
up to that same index instead of duplicating indexing logic:

- QueryString is now parsed and used to build a real restriction, scoped
  to the requested folder. A successful query with zero matches now
  correctly returns "no results" instead of an unfiltered listing; any
  failure to open/query the index (no index yet, corrupt, ...) falls back
  transparently to the previous unrestricted behavior - never a new
  error.
- A plain subject "Contains" (Full string or Prefix) restriction is
  accelerated the same way. Substring mode is deliberately left alone:
  FTS5's token/prefix matching can't reproduce arbitrary mid-word
  substring matches, so accelerating it would silently change semantics.
  Anything else (AND/OR trees, body/attachment search, other properties)
  is untouched and keeps using the original restriction.
- Connections to each user's index.sqlite3 are cached (one per user, for
  the life of the process) instead of opened per FindItem call; opening
  fresh on every call in an "all folders" burst caused real, if
  transient, contention with grommunio-index's periodic regeneration of
  the same file. The cache is capped (256 entries, LRU-evicted) and
  retries a failed open after 5 minutes, so a newly-indexed or
  newly-provisioned mailbox is picked up without a process restart.
- The index base path is configurable (ews.cfg:ews_fts_index_path,
  defaults to /var/lib/grommunio-web/sqlite-index); empty disables the
  acceleration entirely.

Deployed and observed in production for 5 days before this cleanup pass
(config knob, connection-cache hardening, path-safety check) turned it
from a working prototype into something commit-ready.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A8rEks8uGMdavP3zVwnGor
@nourdineb-ops

Copy link
Copy Markdown
Contributor Author

Note on the failing CI check: unrelated to this change — the repo's build workflow has been failing at the install-deps/GnuTLS step (cascading into vmime's CMake configure) since at least the 2026-09-03 push to master itself, not just PRs. Happy to rebase/retrigger once that's sorted.

@nourdineb-ops

Copy link
Copy Markdown
Contributor Author

Following up — a week with no review. Same CI note as above (the install-deps/GnuTLS failure is unrelated to this change).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant