shortbread: allow multi-lingual support

[pnorman]

This converts the name logic to a list of names rather than only en and de. Although currently only en and de are in the list it allows the list to be easily extended when the spec is changed.

[pnorman]

I have most of the street spirit changes done, I just need to review and commit

[1ec5]

From shortbread-tiles/shortbread-docs#68 (comment), it seemed like the discussion was going in the direction of treating name_* as a wildcard rather than enumerating the valid languages directly in the specification. If so, that would shift the editorial decision to implementations like Spirit. Would Spirit hard-code a set of languages based on tagging frequency in OSM, supported interface languages in openstreetmap-website, any valid IETF language tag?

[pnorman]

Would Spirit hard-code a set of languages based on tagging frequency in OSM, supported interface languages in openstreetmap-website, any valid IETF language tag?

Tagging frequency does not work thanks to name:right and similar tags. With this implementation the list of languages has to be hard coded and the intent is to add any languages present in OSM above a very low cutoff. I intend to make a map using local indigenous languages, some of which have under 50 occurrences in the OSM database.

If someone had a function that returns if a string is a valid IETF language tag I could use that but I didn't see any. I also didn't see any comprehensive lists of what is a valid language tag.

The list of languages being >100 long does imply that I need to change the algorithm. Currently it loops through the languages, looking for name tags that match the language. Instead I should loop through tags, looking for name tags and then extracting the language from them to test against a list of languages.

[1ec5]

If someone had a function that returns if a string is a valid IETF language tag I could use that but I didn't see any.

Sure, Planetiler has one. It used to be based on a rudimentary regular expression that covers like 99.997% of OSM data, but for completeness, I deprecated it in onthegomap/planetiler#1404 in favor of running each subkey by Java’s built-in Locale class. This is to support some more exotic subtags that you could easily just not care about, such as name:en-u-sd-usnc=* versus name:en-u-sd-usva=* on this internecine dispute among U.S. states. Many languages have a facility that validates IETF language tags, but there’s nothing like that for Lua. You’d need to port the regular expression to a Lua pattern, or more likely a few of them.

I also didn't see any comprehensive lists of what is a valid language tag.

A whitelist would be impractical, because a language tag can contain so much information that one can mix and match. Given a list of OSM keys in use, you could choose to blacklist certain patterns, such as name:right=* that obviously isn’t a language tag. But a general pattern for what to accept would work quite well enough.

[pnorman]

Sure, Planetiler has one. It used to be based on a rudimentary regular expression that covers like 99.997% of OSM data, but for completeness

Would that consider name:foo a valid IETF tag?

[1ec5]

foo is a well-formed ISO 639 code, and thus a well-formed IETF language tag, but no language is assigned to it. The regular expression accepts it, but the revised function rejects it.

Both the regular expression and its replacement reject noncanonical spellings due to conflict with a different tagging scheme for the name according to some public transport operator. TEC is technically a valid language tag, but the canonical spelling would be tec, so I reject it, as name:TEC=* is actually an obsolete subkey for something else.

[1ec5]

foo is a well-formed ISO 639 code, and thus a well-formed IETF language tag, but no language is assigned to it. The regular expression accepts it, but the revised function rejects it.

I just tried this out in the revised function. Java’s Locale class does accept foo, parsing it as a language code. It does not accept right, because there’s a special check about the language code being 2–3 characters. (Anything 4–8 characters long is only theoretically valid.) I think this is acceptable if your goal is to prevent any unrelated tags from leaking into the localized name part of the schema. name:*=* subkeys are normally interpreted as language tags, so anything else is exceptional. However, if your goal is to minimize the number of lookups, you’ll want to combine this with a list of most common language tags to avoid cruft that nothing on the client side will use.

As a point of reference, OpenHistoricalMap’s tile generator accepts any valid IETF language tag, except that it discards any valid IETF language tag if the subkey has fewer than five occurrences. It still brings in the bogus name:al=* (which should be name:sq=*), but that’s a tagging error, not a problem for the tile generator to concern itself with.

https://github.com/OpenHistoricalMap/ohm-deploy/blob/42482d32dfa6ea848b5fe8f2147256759f06ad98/images/tiler-imposm/queries/utils/fetch_db_languages.sql#L61

[ramSeraph]

If someone had a function that returns if a string is a valid IETF language tag I could use that but I didn't see any. I also didn't see any comprehensive lists of what is a valid language tag.

Is this the only thing blocking adding more languages for the spirit shortbread implementation? If so, I can add a function for it( porting language-tags/langcodes ) in lua. Should that live here or themepark or as a separate luarocks package?

[ramSeraph]

I have put it out in a gist to show how it looks like - https://gist.github.com/ramSeraph/1af8e77fcfd3d800f0b57da44c470cdf. Hope this helps with deciding on how to approach this.

[pnorman]

I don't want to merge this yet because it would require a DB reload and then another DB reload when more languages are added to shortbread.