After digging through repo history, I couldn't find an actual reference for .editPath(), the method producing a sequence of edits represented by the edit-distance calculation.
After further thought and investigation, I realized that there can be more than one possible edit sequence... and that the best approach for future stability is to calculate and return _all_ of them. The trick is to rank them based on the qualities we most look for - and for predictive-text's sliding context window, what we want is a long, unbroken sequence of 'match'-ed tokens, followed by substitutions on the edges.
Previously, I'd finagled the editPath() function to check certain edit types before others, but it always felt hacky and brittle. This new approach seems like a much more robust way forward, even if it probably is a bit more costly to perform.
The new method is implemented separately from the `ClassicalDistanceCalculation` class in order to facilitate tree-shaking. It can easily be imported for local builds or referenced in a temporary class property as needed.
This also has the benefit of _greatly_ simplifying how we compute the tokenized transform indices. Kinda sad I didn't land on this approach the first time. (I actually felt like there _had_ to be something simpler... and it turns out there was!)
With this change, the transform-tokenization methods now indicate which token (relative to the original context's tail token) each of the resulting tokenized transforms affects (and/or creates) in the output tokenization.
Fixes: #14361.
Fixes: #14480.
Relates-to: #14480
This is being done to prepare for a fix to #14480, which is needed to better handle complex input transforms and tokenization scenarios. The easiest way to consistently index the tokenized transforms... is to perform context alignment on the results of transform application.
Prevents scenarios in which a word sliding out of the predictive-text engine's available context window causes the engine's context alignment to fail.
Our cached tokenization data preserves all text for tokens, even those partially slid out of the active context window. We could either adjust/replace the token or allow the leading token to align with notably decreased restrictions. ... or both, to be extra robust (in case of large transform input).
This commit allows alignment with notably decreased restrictions.
I've really been looking forward to figuring this one out. This refactor spins off the bulk of `attemptMatchContext` into a _class_ method on `ContextTokenization` - `.transitionTo` - which uses alignment data to build the new tokenization for incoming context based on the old tokenization in a clear manner.
By factorizing the method in this manner, we pave the way to allow this operation to be performed for different tokenization patterns based on either the initial or the final context (or both!) in the future - something epic-dict-breaker models will likely need to leverage. It also makes the roles of the two halves of the code _far_ clearer than it was before, in my opinion.
