This PR calculates the range of context - possibly comprised of multiple tokens - removed from the prior context and/or appended via incoming input - caused when applying a suggestion. This new approach is much better suited toward operation under whitespace fat-fingering conditions than the original approach.
Build-bot: skip build:web
Test-bot: skip
After using devin.ai to check some of my PRs, it caught something on #15817 that I was able to trace back to some decisions already in place for `TokenResultMapping`. It appears best to simplify the type and its relation to spaceId-tagging now to simplify it, to prevent the bug, and to hopefully prevent re-implementing the bug in the future.
I did consider constructing new instances of SearchNode, just with the changed spaceId... but that may be prone to causing memory churn that can be avoided by just letting spaceId be public.
Build-bot: skip build:web
Test-bot: skip
The implementation of ContextState.analyzeTransition is rather long and large, comprised of multiple subsections. We can get clearer, more maintainable code if we split it into multiple pieces - especially if we can do so in a manner that allows individual pieces to be unit-tested.
Also, with whitespace fat-finger handling coming up, special handling for applied suggestions may be needed. This refactor may facilitate development of a suggestion-specialized variant.
Build-bot: skip build:web
Test-bot: skip
This gives us smaller, more specialized methods - something that's usually a win of its own. It will also help facilitate further adjustments needed for support of whitespace fat-finger prediction & autocorrection.
Build-bot: skip build:web
Test-bot: skip
With the various ways that tokenizations can transition depending upon which potential inputs are applied, it's possible for multiple different tokenizations to transition into the same one. As such, there will no longer be "just one" way that a tokenization is reached.
Accordingly, it's best to perform word-boundary realignment operations (splits, merges) separately from text-editing operations (inserts, deletes).
Build-bot: skip build:web
Test-bot: skip
This function will also be utiliized for specifying tests for the specialized spur types coming up, so rather than duplicate the function, it's best to make it a proper test helper defined outside a specific unit test suite.
As the next work in line will introduce new, specialized spur types designed to replace the current 'legacy spurs', it is wise to clarify existing search-graph unit tests and which sections of the code they actually target. We'll eventually drop behaviors specific to 'legacy' spurs, but those that apply to the new incoming specialized spur types should be preserved.
At the same time, it may be wise to improve the unit testing of each specific type by placing each within its own specialized unit-test suite, then adding new tests that test and clarify the role of each type.
Build-bot: skip build:web
Test-bot: skip
