The new method is implemented separately from the `ClassicalDistanceCalculation` class in order to facilitate tree-shaking. It can easily be imported for local builds or referenced in a temporary class property as needed.
This also has the benefit of _greatly_ simplifying how we compute the tokenized transform indices. Kinda sad I didn't land on this approach the first time. (I actually felt like there _had_ to be something simpler... and it turns out there was!)
With this change, the transform-tokenization methods now indicate which token (relative to the original context's tail token) each of the resulting tokenized transforms affects (and/or creates) in the output tokenization.
Fixes: #14361.
Fixes: #14480.
Relates-to: #14480
This is being done to prepare for a fix to #14480, which is needed to better handle complex input transforms and tokenization scenarios. The easiest way to consistently index the tokenized transforms... is to perform context alignment on the results of transform application.
Prevents scenarios in which a word sliding out of the predictive-text engine's available context window causes the engine's context alignment to fail.
Our cached tokenization data preserves all text for tokens, even those partially slid out of the active context window. We could either adjust/replace the token or allow the leading token to align with notably decreased restrictions. ... or both, to be extra robust (in case of large transform input).
This commit allows alignment with notably decreased restrictions.
I've really been looking forward to figuring this one out. This refactor spins off the bulk of `attemptMatchContext` into a _class_ method on `ContextTokenization` - `.transitionTo` - which uses alignment data to build the new tokenization for incoming context based on the old tokenization in a clear manner.
By factorizing the method in this manner, we pave the way to allow this operation to be performed for different tokenization patterns based on either the initial or the final context (or both!) in the future - something epic-dict-breaker models will likely need to leverage. It also makes the roles of the two halves of the code _far_ clearer than it was before, in my opinion.
