This adds another helper designed to facilitate #14790 and is targeted to eventually replace our current `tokenizeTransform` method. We're not quite ready to drop `tokenizeTransform` and its helpers yet, though.
Build-bot: skip build:web
Test-bot: skip
These windows are designed for use in detecting tokenization shifts both at the start and the end of the context.
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
To be explicit: the method is designed to compare the context after applying
a context transition's edit with the context state after a window slide takes
effect, but before the next transition's edit takes effect.
Build-bot: skip build:web
Test-bot: skip
To be up-front, this is looking ahead significantly to epic/dict-breaker (#12142) concerns. The core idea here: each token should mark which part of the current context it represents - and the best data for the job happens to be the original, true transforms.
With epic/dict-breaker and other possible work that would faciltate considering alternative tokenizations, it is quite possible the original source transform(s) for a tokenization may not actually be represented within that tokenization's input set.
Consider: if the spacebar appears as a fat-finger alternate for a letter key, we may wish to model an alternate tokenization where we actually _used_ that keystroke as a whitespace instead of a standard letter. The resulting whitespace token shouldn't have the original letter key's transform as part of its represented inputs for such a case.
Build-bot: skip build:web
Test-bot: skip
Tokens in the active context should not be transposable, so this commit adds an option that permits disabling transpose edits during edit-distance calculation.
Build-bot: skip build:web
Test-bot: skip
The main goal of this commit is to allow computeDistance to operate properly with both ClassicalDistanceCalculation and SegmentableDistanceCalculation. For now, context alignment via computeAlignment will continue to use the former edit-distance calculation type, but the goal is to convert to use of the latter and its support for 'split' / 'merge' edit operations in the near future.
Build-bot: skip build:web
Test-bot: skip
After a bit of thought, I realized that it's feasible to support certain kinds of "partial" splits and edits. If the only missing portion is entirely on one end or the other of the "combined" token, that isn't costly to compute - and it very much matches the specific cases that motivated 'merge' / 'split' support to begin with.
Suppose the word `can't` is partially in the context, at the start of the sliding context window. Sliding backward one character at a time and using our default wordbreaker, we'd get...
- `t`
- `'`, `t`
- `n't`
`n't` is a "partial merge" scenario - we're missing the 'n', but the rest of the pieces are present. The new changes allow us to detect that `'` and `t` combine as part of the `n't` token.
Note that this still requires a 1-to-many relationship between the "input" and "match" string sequences; there will be no support for a hypothetical "merge-then-split" operation. (Say, combining 2 and splitting the result into 3 - or vice versa.)
This change will provide a way to align more directly from edit path data, rather than rely on the post-op guesswork that's been used until now to approximate it.
