ews: accelerate FindItem search via grommunio-web's FTS5 index - #341
Open
nourdineb-ops wants to merge 2 commits into
Open
nourdineb-ops wants to merge 2 commits into
nourdineb-ops wants to merge 2 commits into
Conversation
Outlook's Instant Search box sends its query as the QueryString (AQS) element on FindItem, which gromox never parsed - the element existed only as a schema comment, so the search term was silently dropped and the folder's unfiltered contents were returned and displayed as "results". Subject "Contains" searches went through gromox's own content-restriction evaluation, a per-row strstr()/strcasestr() scan with no text index, making an "all folders" search from Outlook slow (one FindItem per folder in a burst, each doing a full unindexed scan). grommunio-web already maintains a per-user SQLite FTS5 index (via the grommunio-index package/timer) for its own search UI. This wires FindItem up to that same index instead of duplicating indexing logic: - QueryString is now parsed and used to build a real restriction, scoped to the requested folder. A successful query with zero matches now correctly returns "no results" instead of an unfiltered listing; any failure to open/query the index (no index yet, corrupt, ...) falls back transparently to the previous unrestricted behavior - never a new error. - A plain subject "Contains" (Full string or Prefix) restriction is accelerated the same way. Substring mode is deliberately left alone: FTS5's token/prefix matching can't reproduce arbitrary mid-word substring matches, so accelerating it would silently change semantics. Anything else (AND/OR trees, body/attachment search, other properties) is untouched and keeps using the original restriction. - Connections to each user's index.sqlite3 are cached (one per user, for the life of the process) instead of opened per FindItem call; opening fresh on every call in an "all folders" burst caused real, if transient, contention with grommunio-index's periodic regeneration of the same file. The cache is capped (256 entries, LRU-evicted) and retries a failed open after 5 minutes, so a newly-indexed or newly-provisioned mailbox is picked up without a process restart. - The index base path is configurable (ews.cfg:ews_fts_index_path, defaults to /var/lib/grommunio-web/sqlite-index); empty disables the acceleration entirely. Deployed and observed in production for 5 days before this cleanup pass (config knob, connection-cache hardening, path-safety check) turned it from a working prototype into something commit-ready. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8rEks8uGMdavP3zVwnGor
Contributor
Author
|
Note on the failing CI check: unrelated to this change — the repo's |
Contributor
Author
|
Following up — a week with no review. Same CI note as above (the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outlook's Instant Search box sends its query as the
QueryString(AQS) element onFindItem, which gromox never parsed — the element existed only as a schema comment, so the search term was silently dropped and the folder's unfiltered contents were returned and displayed as "results". SubjectContainssearches went through gromox's own content-restriction evaluation, a per-rowstrstr()/strcasestr()scan with no text index, making an "all folders" search from Outlook slow (oneFindItemper folder in a burst, each doing a full unindexed scan).grommunio-webalready maintains a per-user SQLite FTS5 index (via thegrommunio-indexpackage/timer) for its own search UI. This wiresFindItemup to that same index instead of duplicating indexing logic:QueryStringis now parsed and used to build a real restriction, scoped to the requested folder. A successful query with zero matches now correctly returns "no results" instead of an unfiltered listing; any failure to open/query the index (no index yet, corrupt, ...) falls back transparently to the previous unrestricted behavior — never a new error.Contains(Full string or Prefix) restriction is accelerated the same way. Substring mode is deliberately left alone: FTS5's token/prefix matching can't reproduce arbitrary mid-word substring matches, so accelerating it would silently change semantics. Anything else (AND/OR trees, body/attachment search, other properties) is untouched and keeps using the original restriction.index.sqlite3are cached (one per user, for the life of the process) instead of opened perFindItemcall; opening fresh on every call in an "all folders" burst caused real, if transient, contention withgrommunio-index's periodic regeneration of the same file. The cache is capped (256 entries, LRU-evicted) and retries a failed open after 5 minutes, so a newly-indexed or newly-provisioned mailbox is picked up without a process restart.ews.cfg:ews_fts_index_path, defaults to/var/lib/grommunio-web/sqlite-index); empty disables the acceleration entirely.Deployed and observed in production for several days before this cleanup pass (config knob, connection-cache hardening, path-safety check) turned it into something PR-ready.
🤖 Generated with Claude Code