Skip to content

Add exact-count full-text coverage to Big5 - #832

Merged
OVI3D0 merged 2 commits into
opensearch-project:mainfrom
OVI3D0:codex/big5-full-text-count-coverage
Sep 25, 2026
Merged

OVI3D0 merged 2 commits into
opensearch-project:mainfrom
OVI3D0:codex/big5-full-text-count-coverage

Conversation

@OVI3D0

@OVI3D0 OVI3D0 commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Description

Adds five exact-count full-text operations to the default Big5 test procedure:

  • high-frequency Boolean AND and OR counts over the message field
  • prefix, wildcard, and regexp counts over message patterns selected from the live Big5 term dictionary

Each operation uses size: 0, track_total_hits: true, and request_cache=false so the benchmark exercises exhaustive collection instead of top-k pruning or cached whole responses. This fills the routine performance coverage gap highlighted by opensearch-project/OpenSearch#23117, including bulk postings iteration and postings reuse during multi-term expansion.

The multi-term patterns each expand beyond the 1,024 Boolean-clause limit per shard when inspected with a scoring Boolean rewrite, confirming that they exercise substantial term expansion.

Issues Resolved

Related to opensearch-project/OpenSearch#23117 and opensearch-project/OpenSearch#23107.

Testing

  • New functionality includes testing

Local validation:

  • opensearch-benchmark list workloads --workload-path <checkout>/big5 --distribution-version 3.8.0
  • opensearch-benchmark info --workload-path <checkout>/big5 --test-procedure big5 --include-tasks high-frequency-and-count,high-frequency-or-count,prefix-count,wildcard-count,regexp-count

Full-corpus validation:

  • Ran OSB against an existing Amazon OpenSearch Service 3.5 domain containing the complete 116,000,000-document Big5 index.

Backport to Branches:

  • 6
  • 7
  • 1
  • 2
  • 3

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off my commits, see the OpenSearch contributing documentation.

Signed-off-by: Michael Oviedo <mikeovi@amazon.com>
Signed-off-by: Michael Oviedo <mikeovi@amazon.com>
@OVI3D0
OVI3D0 marked this pull request as ready for review September 24, 2026 22:47
@OVI3D0
OVI3D0 merged commit 97fd1d2 into opensearch-project:main Sep 25, 2026
4 checks passed
@OVI3D0
OVI3D0 deleted the codex/big5-full-text-count-coverage branch September 25, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants