Port fix for Token's positionLength from Lucene 4.8.1, #1381

[paulirwin]

• You've read the Contributor Guide and Code of Conduct.
• You've included unit or integration tests for your change, where applicable.
• You've included inline docs for your change, where applicable.
• There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Backport fix for Token's positionLength from Lucene 4.8.1

Partial #1381 (Batch 3)

Description

This backports commit c921f8a009c for LUCENE-5639 from the 4.8.1 release. This fixes usage of Token's positionLength value.

[NightOwl888]

Looks like a faithful port of the upstream commit.

I did not check what all of the implications of adding this commit will be, but did ask ChatGPT whether changing GetHashCode() and Equals() will affect binary compatibility of the index and the answer was generally no. But we should probably merge #1351 and rebase this against it to be sure.

[paulirwin]

But we should probably merge #1351 and rebase this against it to be sure.

Since #1351 had some significant feedback that I addressed, I didn't want to dismiss your review... instead I merged this one and rebased #1351, and the tests still pass, which accomplishes the same concern.