bug: khmer model custom wordbreaker issues

[MakaraSok]

Describe the bug

The crash happened after this activity was done. See the crash in action:

predictive.text.crashes.mov

Reproduce the bug

No response

Expected behavior

No response

Related issues

No response

Keyman apps

• Keyman for Android
• Keyman for iPhone and iPad
• Keyman for Linux
• Keyman for macOS
• Keyman for Windows
• Keyman Developer
• KeymanWeb
• Other - give details at bottom of form

Keyman version

17.0.104-alpha

Operating system

iOS 16.4

Device

iPhone Pro Max Simulator

Target application

No response

Browser

No response

Keyboard name

sil_jarai

Keyboard version

1.0

Language name

Jarai

Additional context

https://keyman.com/keyboards/sil_jarai?bcp47=jra-khmr

[jahorton]

This is essentially the same issue seen at keymanapp/keyman#6900, but conflated with issues that arise when handling Khmer script.

Relevant codeblock from the corresponding lexical model:

lexical-models/release/sil/sil.jra-khmr.jarai/source/sil.jra-khmr.jarai.model.ts

Lines 12 to 21 in 0491eb0

	wordBreaker: function(str: string) {
	return str.split(/\s/).map(function(token) {
	return {
	left: str.indexOf(token),
	start: str.indexOf(token),
	right: str.indexOf(token) + token.length,
	end: str.indexOf(token) + token.length,
	text: token
	}
	});

  wordBreaker: function(str: string) {
    return str.split(/\s/).map(function(token) {
      return {
        left: str.indexOf(token),
        start: str.indexOf(token),
        right: str.indexOf(token) + token.length,
        end: str.indexOf(token) + token.length,
        text: token
      }
    });

For starters, note that this "wordbreaker" was always intended to be something of a stand-in until we develop a better way to handle cases with scripts that don't normally do wordbreaking. (The majority language for the script is Khmer, which doesn't... even if Jarai itself does.)

Furthermore, this wordbreaker is not aware of any implicit meaning behind any punctuation marks in the script - it only breaks on spaces and nothing else. Thus, the guillemets (the double angle-brackets acting as quotation marks) are considered the same as letters and thus part of the same word.

Refer to the video associated with keymanapp/keyman#6900:

lm.replace.quote.and.character.typed.with.the.selected.suggestion.mov

The guillemets are replaced because, as far as the system knows, they are part of the word, not separate. This, in turn, naturally has a strong knock-on effect of making predictions a lot more difficult. No Khmer word actually starts with a left-guillemet («), after all.

With my current attempts at reproducing it, the engine actually does recover on the first post-guillemet keystroke most of the time. Selecting such a suggestion also erases the guillemet due to the details noted above re: the model's wordbreaker. It also recovers instantly when starting a new word. Thus, it's not "crashing" - just "failing to find any suggestions."

Finally, note that the predictive-text engine will only allow so much corrections before it stops looking. Having to outright delete the « in order to make good suggestions for the text after it is quite costly, and that doesn't reset within the word at present. So, even when "working", corrections will seem markedly more limited in this context.

[jahorton]

Looking back through related issue and PR history, this thread seems particularly relevant: https://github.com/keymanapp/keyman/pull/6574/files#r861500917

If we did allow character-class overrides, that'd provide a way to avoid writing a complex custom wordbreaker. But, for now, perhaps I should just tweak this hacky would-be wordbreaker to hack off the « from the actual word.