Backport remaining Lucene 4.8.1 bug fixes #1381

Is there an existing issue for this?

• I have searched the existing issues

Task description

We have effectively been targeting 4.8.1 instead of 4.8.0 for a while now, so we might as well close that gap before exploring #1293. This release includes some worthwhile bugfixes.

What follows is the analysis by Claude Code against the Lucene 4.8.1 source code and git history. There may be errors/omissions.

Background

Lucene.NET targets Lucene 4.8.1, but an audit of the upstream 4.8.0 → 4.8.1
diff (the 16 LUCENE-* bug fixes in lucene/CHANGES.txt) found that several
fixes were never ported. No existing issue tracks these. #1293 is explicitly
scoped to fixes after 4.8.1 (4.9-4.10), so this gap is currently untracked.

The fixes below match pre-fix 4.8.0 behavior in current master. They are
grouped into batches that can be broken out into sub-issues. Several are
trivial; one (LUCENE-5628) fixes a real StackOverflowException that is
already biting the port (AnalyzingSuggesterTest.TestTooLongSuggestion
sits commented out with a "currently produces a stack overflow" note).

Already ported (no action)

LUCENE-5662, LUCENE-5599/5600, LUCENE-5653, LUCENE-5559, LUCENE-5644,
LUCENE-5671 are all present, with matching tests for 5653, 5559, and 5599/5600.

Remaining work

Batch 1 — Trivial production fixes (1-5 lines each)

PR: #1413

• LUCENE-5635: move stream.Reset() inside the try block in
  DocInverterPerField.cs (~line 107) so an exception on reset is handled.
• LUCENE-5668: change eligible.Count >= allowedSegCountInt to > in
  TieredMergePolicy.cs (~line 415); port the testUnbalancedMergeSelection
  test.
• LUCENE-5656: move dvGens.Add(gen) inside the per-gen loop in
  SegmentReader.cs (~lines 197-209) instead of AddRange after the loop
  (rare fd leak on docvalues-update exception).
• LUCENE-5644 follow-up: add the missing PulseAll/notifyAll at the
  end of DeactivateUnreleasedStates() in DocumentsWriterPerThreadPool.cs.

Batch 2 — LUCENE-5641 (RateLimiter)

PR: #1420

• Rewrite RateLimiter.cs: add MinPauseCheckBytes, synchronize Pause,
  drop nsPerByte/lastNS and the bytes == 1 shortcut.
• Update RateLimitedIndexOutput.cs to accumulate bytesSinceLastPause
  and only call Pause when it exceeds MinPauseCheckBytes.
• Port the multithreaded TestRateLimiter.testThreads.

Fixes the "silently rate-limits at 8 MB/sec regardless of configured rate" bug.

Batch 3 — LUCENE-5639 (Token.PositionLength)

PR: #1421

• Wire positionLength into all 9 sites in Token.cs: Clear, Clone,
  Equals, GetHashCode, ClearNoTermBuffer, the private Reinit, the three
  Reinit(Token …) overloads, CopyTo, and ReflectWith. (The field/property
  exist but none of the handling is wired up.)
• Add the positionLength attribute-map entry in TestToken.

Batch 4 — LUCENE-5628 + LUCENE-5660 (highest priority)

PR: #1422

• LUCENE-5628: make GetFiniteStrings iterative (PathNode stack) in
  SpecialOperations.cs instead of recursive (one stack frame per character).
• LUCENE-5660: re-enable the suggester guard in AnalyzingSuggester.cs
  (currently Debugging.Assert(SpecialOperations.IsFinite(...)) at ~line 982).
• OfflineSorter: add the len > short.MaxValue guard in
  ByteSequencesWriter.Write (OfflineSorter.cs).
• Add the GetFiniteStringsRecursive helper to AutomatonTestUtil.cs.
• Port the full 4.8.1 TestSpecialOperations suite and un-comment
  AnalyzingSuggesterTest.TestTooLongSuggestion (AnalyzingSuggesterTest.cs
  ~line 1502).

Resolves an active StackOverflowException already stubbed out in the suite.

Batch 5 — LUCENE-5654 + test infrastructure (largest; gated by infra)

PR: #1424

• Production: convert the 4 close() call sites to the success-boolean
  pattern: CompoundFileDirectory, CompoundFileWriter, Directory.Copy, and
  FSDirectory output close. (Keep the existing
  DisposeWhileHandlingException(priorException, …) IOUtils overloads — only
  the call sites change.)
• Test framework prerequisite: port the entire cranky codec package
  (9 files: CrankyCodec + CrankyDocValues/FieldInfos/LiveDocs/Norms/ Postings/SegmentInfo/StoredFields/TermVectorsFormat). Only CrankyTokenFilter
  is ported today.
• Tests (blocked on the two prerequisites above): port
  TestIndexWriterExceptions2, TestIndexWriterOutOfMemory, and
  TestIndexWriterThreadsToSegments (the last validates LUCENE-5644).
• LUCENE-5644 follow-up (011a35b0a, no JIRA): add active-thread-state counting + infoStream message to FindLargestNonPendingWriter in FlushPolicy.cs (~line 131) and prefix the FlushByRamOrCountsPolicy.cs flush message with "trigger " (~line 100).

Suggested sequencing

Batches 1-4 are independent and deliver all user-facing production fixes;
Batch 4 first since it resolves an active stack-overflow. Batch 5 is a
larger infra investment whose production payload (5654) is lower-risk, so it's
a reasonable last step.