[rishabhmaurya]

Description

When a query delegates a filter predicate to Lucene, the per-segment match
bitmap was built once per row group: every RG re-ran createCollector → weight.scorer(leaf). For an expensive query — e.g. a leading-wildcard
WildcardQuery over a large term dictionary — scorer() walks the whole FST,
and that cost is independent of how many docs match. With N row groups in a
segment spread across N partitions, one segment paid N redundant FST walks.

This change builds the bitmap once per segment and shares it across every
partition and chunk via an Arc<OnceLock>; each RG slices its own window out
of the shared segment-relative bitmap. Both delegation roles are covered:

• correctness leaves — per-segment cache plus a concurrent warm_cache()
  prewarm on the blocking pool, so the bitmap is ready (or in flight) before
  the first poll.
• performance peers — per-(annotation_id, writer_generation) cache, built
  lazily on the first RG that passes the selectivity gate.

Measured on ClickBench q23 (single-segment shard): index_query_time drops
from ~21s to ~2.2s. The selectivity gate and page/bloom pruning are unchanged —
a segment whose RGs never pass the gate still builds nothing.

Failure semantics. The cached cell holds a Result, not an Option, so a
drained-but-empty segment (genuinely zero matches) is distinct from a failed
drain. A correctness predicate has no DataFusion fallback, so a failed drain
propagates as an error rather than silently dropping the row group; a
performance peer ignores the failure and lets DataFusion's FilterExec
evaluate the retained predicate. Peer and correctness caches share one
SegmentBitmapCache<K> helper.

Kill-switch. Gated by the dynamic cluster setting
datafusion.indexed.bitmap_cache_enabled (default true). Set it false to
fall back to the legacy per-row-group path (eager correctness collector; peers
skip precompute) without a redeploy.

A per-chunk OnceLock variant and a K-way parallel drain were both tried and
reverted: concurrent createCollector calls for one Lucene segment serialize
on Java-side FST/term-enum state, so splitting the drain is slower than a single
serial pass.

Related Issues

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[rishabhmaurya]

Retargeting against opensearch-project/OpenSearch:main as a draft.