After changes added in #14880, one of our unit tests failed. After lots of investigation, it turns out that the `isSubstitutionAlignable` does not catch certain cases that `determineContextSlideTransform` does. The latter does need some improvement to work as a fix, but a small change lets it pick up the slack and replace the former.
As it's also used during context-state transition, re-using the same method in both places gives us far more confidence that we won't have an error the second time around.
Build-bot: skip build:web
Test-bot: skip
Build-bot: skip test:web
Test-bot: skip
The new transform-tokenization "similarity" check added in #14822 is more limiting when only one tokenization is supported. As an interim measure, this commit and PR are designed to be reverted once the engine can handle multiple tokenizations properly and thus can swap fully to use of the new keying strategy.
This method is designed to identify changes in tokenization upon input that result in the same tokenization effects when applied. Such cases are safe to batch together when building search path extensions.
Build-bot: skip build:web
Test-bot: skip
This adds another helper designed to facilitate #14790 and is targeted to eventually replace our current `tokenizeTransform` method. We're not quite ready to drop `tokenizeTransform` and its helpers yet, though.
Build-bot: skip build:web
Test-bot: skip
These windows are designed for use in detecting tokenization shifts both at the start and the end of the context.
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
To be explicit: the method is designed to compare the context after applying
a context transition's edit with the context state after a window slide takes
effect, but before the next transition's edit takes effect.
Build-bot: skip build:web
Test-bot: skip
To be up-front, this is looking ahead significantly to epic/dict-breaker (#12142) concerns. The core idea here: each token should mark which part of the current context it represents - and the best data for the job happens to be the original, true transforms.
With epic/dict-breaker and other possible work that would faciltate considering alternative tokenizations, it is quite possible the original source transform(s) for a tokenization may not actually be represented within that tokenization's input set.
Consider: if the spacebar appears as a fat-finger alternate for a letter key, we may wish to model an alternate tokenization where we actually _used_ that keystroke as a whitespace instead of a standard letter. The resulting whitespace token shouldn't have the original letter key's transform as part of its represented inputs for such a case.
Build-bot: skip build:web
Test-bot: skip
Tokens in the active context should not be transposable, so this commit adds an option that permits disabling transpose edits during edit-distance calculation.
Build-bot: skip build:web
Test-bot: skip
