To be up-front, this is looking ahead significantly to epic/dict-breaker (#12142) concerns. The core idea here: each token should mark which part of the current context it represents - and the best data for the job happens to be the original, true transforms.
With epic/dict-breaker and other possible work that would faciltate considering alternative tokenizations, it is quite possible the original source transform(s) for a tokenization may not actually be represented within that tokenization's input set.
Consider: if the spacebar appears as a fat-finger alternate for a letter key, we may wish to model an alternate tokenization where we actually _used_ that keystroke as a whitespace instead of a standard letter. The resulting whitespace token shouldn't have the original letter key's transform as part of its represented inputs for such a case.
Build-bot: skip build:web
Test-bot: skip
Tokens in the active context should not be transposable, so this commit adds an option that permits disabling transpose edits during edit-distance calculation.
Build-bot: skip build:web
Test-bot: skip
The main goal of this commit is to allow computeDistance to operate properly with both ClassicalDistanceCalculation and SegmentableDistanceCalculation. For now, context alignment via computeAlignment will continue to use the former edit-distance calculation type, but the goal is to convert to use of the latter and its support for 'split' / 'merge' edit operations in the near future.
Build-bot: skip build:web
Test-bot: skip
After a bit of thought, I realized that it's feasible to support certain kinds of "partial" splits and edits. If the only missing portion is entirely on one end or the other of the "combined" token, that isn't costly to compute - and it very much matches the specific cases that motivated 'merge' / 'split' support to begin with.
Suppose the word `can't` is partially in the context, at the start of the sliding context window. Sliding backward one character at a time and using our default wordbreaker, we'd get...
- `t`
- `'`, `t`
- `n't`
`n't` is a "partial merge" scenario - we're missing the 'n', but the rest of the pieces are present. The new changes allow us to detect that `'` and `t` combine as part of the `n't` token.
Note that this still requires a 1-to-many relationship between the "input" and "match" string sequences; there will be no support for a hypothetical "merge-then-split" operation. (Say, combining 2 and splitting the result into 3 - or vice versa.)
This change will provide a way to align more directly from edit path data, rather than rely on the post-op guesswork that's been used until now to approximate it.
This commit adds unit tests for the methods extracted in #14716 designed to help automate testing for work done in #14700 and #14537, among other recent epic/autocorrect PRs.
There's also a very small bit of cleanup of logic that affected one of the new tests.
Build-bot: skip test:web
Test-bot: skip
This prevents display errors associated with reversions that occurred once the worker started supplying reversions as part of prediction requests.
Also hides suggestions / reversions that depend on context states that have aged out of the Web engine's cache.
In particular, this merges in the recent changes
to `master` from #14618 that adjust the
edit-distance calculation class and simplify its
input format.
These changes intersected with some epic/autocorrect
code added by #14617 and #14462, which needed tweaks
to work correctly with the new format.
