Add bidirectional index/codec compatibility harness for Lucene, #270

[paulirwin]

• You've read the Contributor Guide and Code of Conduct.
• You've included unit or integration tests for your change, where applicable.
• You've included inline docs for your change, where applicable.
• There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Add bidirectional index/codec compatibility harness for Lucene

Fixes #270

Description

Adds a two-way check that Lucene.NET and Apache Lucene 4.8.1 (Java) can each read an index written by the other. A new self-contained Maven project under src/java/index-compat generates and reads indexes; the .NET side is covered by TestJavaCompatibility in the _I-J test shard.

Both runtimes write the same deterministic document set (mirroring the schema in TestBackwardsCompatibility), then the other runtime opens it, runs CheckIndex (the codec integrity gate over the per-file checksums), and asserts the contents match. No byte-for-byte comparison is done, since some header fields (e.g. java.vendor) legitimately differ between runtimes.

No index fixtures are committed: every index is generated fresh into the gitignored work/ folder. run-compat.sh / run-compat.ps1 drive both directions end to end and return a non-zero exit code on failure. In .NET a missing compat index makes the cross-runtime test inconclusive (i.e. if you just cloned the code and haven't run the compat script yet); in Java a missing .NET index fails the test. The .NET writer forces the real Lucene46Codec, since LuceneTestCase otherwise randomizes Codec.Default to test-only impostors a stock Lucene cannot load.

Also adds a hand-maintained GitHub Actions workflow (not generated by Generate-TestWorkflows.ps1, since it needs a JDK and a Maven <-> dotnet orchestration) that runs both directions on a matrix of ubuntu/windows x net10.0/net8.0/net472/net48. Each .NET step asserts "Passed: 1" so a skipped or filtered-out test fails the job instead of going green silently. The workflow uses its own 'compat-' prefixed NuGet cache key to avoid colliding with the generated workflows' full-solution cache.

AI: Written with Claude Code, Opus 4.8, and reviewed/edited by hand.

[Copilot]

Pull request overview

Adds a bidirectional Lucene.NET ↔ Apache Lucene 4.8.1 (Java) index/codec compatibility harness. The harness deterministically generates equivalent indexes in both runtimes, validates them via CheckIndex, and semantically verifies document/content/docvalues parity, with scripts and CI orchestration to run both directions.

Changes:

• Added a shared deterministic document/index contract (CompatDocs) and .NET tests to (a) round-trip locally, (b) read Java-generated indexes, and (c) write .NET indexes for Java to read.
• Added a self-contained Maven project (src/java/index-compat) that writes Java indexes and JUnit-tests reading .NET indexes.
• Added cross-platform driver scripts plus a dedicated GitHub Actions workflow to execute both directions on an OS/TFM matrix.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/Lucene.Net.Tests/Support/Index/TestJavaCompatibility.cs	.NET-side tests for round-trip, Java→.NET read, and .NET→Java write, including semantic verification.
src/Lucene.Net.Tests/Support/Index/CompatDocs.cs	Deterministic shared document set and index writer logic for the cross-runtime contract.
src/java/index-compat/src/test/java/org/apache/lucenenet/compat/TestDotNetCompatibility.java	JUnit test that validates Lucene 4.8.1 can read a Lucene.NET-written index and contents match contract.
src/java/index-compat/src/main/java/org/apache/lucenenet/compat/GenerateIndex.java	Java CLI entry point to generate Java-written compatibility indexes.
src/java/index-compat/src/main/java/org/apache/lucenenet/compat/CompatDocs.java	Java mirror of the deterministic contract and index writer / verifier helpers.
src/java/index-compat/run-compat.sh	Bash entry-point wrapper that invokes the PowerShell driver.
src/java/index-compat/run-compat.ps1	Cross-platform orchestration script to run both directions end-to-end.
src/java/index-compat/run-compat.bat	Windows CMD wrapper for invoking the PowerShell driver.
src/java/index-compat/README.md	Documentation for running the harness and explaining the contract.
src/java/index-compat/pom.xml	Maven project definition for Lucene 4.8.1 compatibility harness and JUnit execution.
src/java/index-compat/mvnw.cmd	Maven wrapper (Windows).
src/java/index-compat/mvnw	Maven wrapper (POSIX).
src/java/index-compat/.mvn/wrapper/maven-wrapper.properties	Maven wrapper configuration.
.rat-excludes	Excludes Maven wrapper artifacts from RAT checks.
.gitignore	Ignores Java build outputs and harness-generated work/ directories.
.github/workflows/Lucene-Net-Index-Compatibility.yml	Dedicated CI workflow to run both directions across OS/TFM matrix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[NightOwl888]

I fixed the merge commit, rebased against the latest master, and force pushed the branch. So, be sure to reset your branch if you run it locally again.

I discovered that these files have several references to specific versions of Lucene and Lucene.NET that are hard coded in many different places. The Lucene files strictly avoid exact versions because it means lots of maintenance every time a new version is released. Also, we are testing a codec, not a specific Lucene version, so we should reflect that in our file naming conventions. I went through and picked out some examples in the code, but this also applies to documentation and comments. These should all be version-agnostic and tied to specific constants that update with Lucene version and/or codec version so we don't have to touch these files to update version numbers, file names, and documentation. Note that recent versions of Lucene no longer have a LUCENE_VERSION_CURRENT constant, but whatever approach that they are using to replace it is what we will need to use going forward.