This method is designed to identify changes in tokenization upon input that result in the same tokenization effects when applied. Such cases are safe to batch together when building search path extensions.
Build-bot: skip build:web
Test-bot: skip
This adds another helper designed to facilitate #14790 and is targeted to eventually replace our current `tokenizeTransform` method. We're not quite ready to drop `tokenizeTransform` and its helpers yet, though.
Build-bot: skip build:web
Test-bot: skip
These windows are designed for use in detecting tokenization shifts both at the start and the end of the context.
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
To be explicit: the method is designed to compare the context after applying
a context transition's edit with the context state after a window slide takes
effect, but before the next transition's edit takes effect.
Build-bot: skip build:web
Test-bot: skip
To be up-front, this is looking ahead significantly to epic/dict-breaker (#12142) concerns. The core idea here: each token should mark which part of the current context it represents - and the best data for the job happens to be the original, true transforms.
With epic/dict-breaker and other possible work that would faciltate considering alternative tokenizations, it is quite possible the original source transform(s) for a tokenization may not actually be represented within that tokenization's input set.
Consider: if the spacebar appears as a fat-finger alternate for a letter key, we may wish to model an alternate tokenization where we actually _used_ that keystroke as a whitespace instead of a standard letter. The resulting whitespace token shouldn't have the original letter key's transform as part of its represented inputs for such a case.
Build-bot: skip build:web
Test-bot: skip
Tokens in the active context should not be transposable, so this commit adds an option that permits disabling transpose edits during edit-distance calculation.
Build-bot: skip build:web
Test-bot: skip
The main goal of this commit is to allow computeDistance to operate properly with both ClassicalDistanceCalculation and SegmentableDistanceCalculation. For now, context alignment via computeAlignment will continue to use the former edit-distance calculation type, but the goal is to convert to use of the latter and its support for 'split' / 'merge' edit operations in the near future.
Build-bot: skip build:web
Test-bot: skip
