Some more bibliography improvements

[Omikhleia]

Work in progress - progressively back-porting upstream some things from my (private) fork of the bibliography components.

An Overview Of The Changes So Far...

(Slightly) Breaking changes

• The legacy bibliography implementation is removed.

  The CSL-based bibliography engine, introduced in SILE 0.15.7 at the end of Nov. 2024, was foreshadowing the eventual deprecation of the earlier home-grown legacy implementation of (a very limited and wrong at places subset) of the Chicago author-date style.

  It's now the only one supported, so the code base can be simplified and extended with new features (see below).

  This also encompasses the removal of the i18n keys (in language files) that were used by the legacy implementation.

Features

• \printbibliography supports new cool options:

  • filter: A space separated list of filters to apply to the bibliography, when cited is false.

    It allows filtering by entry type (ex. type-book or not-type-book), by keywords (ex. keyword-linguistics), or issue date (ex. issued-2020 for entries issued in 2020, or issued-2023-2025 for entries issued between 2023 and 2025).

    Entry types and keywords are to be understood in terms of CSL here (not BibTeX).

    Besides those simple built-in filters (which, in my experience at least, cover most use cases), a (very and somewhat internal) low-level Lua API is provided to allow users to implement their own named filters.

    (The reasons for these filters being allowed only when cited is false are that they are meant to filter the complete bibliography. Filtering cited entries would make tracking and resetting more complex, for little gain in normal use cases, in my opinion - at least for now, with things still in the flux.)

  • related: A boolean option to include "related" entries in the bibliography (in an indented block under the main entries).

    With a Bib(La)TeX bibliography, it correspond to entries which have a related field (a comma-separated list of entry keys). So, say, if you have a book B and an article A (e.g. possibly a review, commentary, etc.) with a related field containing the key of B, A will be listed nested under B in the bibliography (independently of being cited or not, on its own).

• By default, ISBNs and ISBNs in bibliographies can now break at dashes, which is useful for justification and line breaking (as these elements are quite long).

  This can be disabled by setting the breakISBN option to false in the bibliography options.

• Bib(La)TeX support improvements:

  • The ids field for citation key aliasing is now supported.

  • Several fields (ids, keywords, related, and xdata) are interpreted as lists of comma-separated values, as they should be.

• \nocite elements are now accepted in \cites constructs for grouped citation, e.g., \cites{\cite[key1]\nocite[\key2] -- it's a convenience, allowing one to "skip" entries in grouped citations, while keeping them in the
  bibliography of cited works, without much rewriting of that block.

Fixes

• Sorting is more consistent in its attempt at a strict weak ordering, avoiding random exceptions.

• Superscript "folding" is now applied to ordinal numbers in bibliographies, as it was already done for other terms.

• State is better handled across multiple bibliography runs, avoiding issues with author substitution (e.g., dashes) in per-chapter bibliographies (or when the new "related" references are enabled).

• Smart straight quotes transformation plays better with italic text (when this extension is enabled, which is the default).

• Rework how sorting and substitutions are done in bibliographies. This is an attempt at fixing Bibliography CSL "substitute" element does not always work as expected #2283 but it still seems that something is amiss with certain styles. Well, it might be partial, either due to a misinterpretation of the specification or a real bug. I haven't been able to pinpoint it, despite hours of attempts 😿 -- But it does repair something that I broke in SILE 0.15.13, and the code should be much better as new refactored.

Other changes

• Welcome the new CslProcessor class.

  It's a big refactor, splitting most of the CSL processing logic out of the bibtex SILE package, which now just implements the command "layer" for SILE as a typesetting engine, delegating the work to that processor for the bibliography processing.

  This also allows using the CSL processing logic outside of SILE's regular processing flow (e.g., using SILE as a "Lua-on-steroids" toolkit with ICU, etc. but without going through the typesetter, PDF output, etc.)¹

• Improved in-code documentation, with better LDoc annotations.

Closes #2126

Closes #2085

Regarding #2283 - the changes here might be a partial / imperfect fix, as noted above, but the code makes more sense now.

Footnotes

1. For instance, I am using it to generate HTML versions of my bibliographies, soon to be released. It's still a bit rough, but it paves the way for better abstractions. ↩

[Omikhleia]

For the record:
My ETA for my work on this topic is around end of August 2025 (where I'll use the new features and fixes for a new edition of one of my books). I'm mostly done (and could have moved the PR to "Ready", but I have a short break the two next weeks, and will check this and other topics upon my return.

[Omikhleia]

My ETA for my work on this topic is around end of August 2025

Main objectives achieved, and PR moved to Ready.

(I had two other very secondary objectives in my wish-list. I'll probably open dedicated issues for them. This PR here is already a nice baby.)

(Le Dragon de Brume will soon communicate regarding a book update, and an online web site. Both were made with SILE, and the code from this PR.)

I am suggesting this should go in 0.15.14, nearly one year after the work that eventually went in 0.15.7 was initiated. True, it's a slightly breaking assuming some people might have relied on the legacy bibtex implementation in anything serious... I cannot believe such users really exist; and even so, the update should be quite transparent, or at least easy and understandable.