Joshua Horton
aa477a9ee3
fix(web): restore unit tests
2025-10-07 11:30:13 -05:00
Joshua Horton
604b2526a1
refactor(web): use tokenization analysis to maintain delayed reversion feature
2025-10-07 11:29:47 -05:00
Joshua Horton
b2cb13741a
refactor(web): rework preservation transforms to prep for multi-tokenization
...
Build-bot: skip build:web
Test-bot: skip
2025-10-07 11:25:13 -05:00
Joshua Horton
c3e5c66a6e
docs(web): thoughts toward upcoming changes for preservation transforms
2025-10-07 11:19:15 -05:00
Joshua Horton
39dd3b50a3
change(web): clean up computation of preservationTransform
2025-10-07 09:20:37 -05:00
Joshua Horton
0cf7930cad
change(web): remove old transitionTo method in favor of new tokenization-transition method
2025-10-03 08:28:07 -05:00
Joshua Horton
49af0c18a5
feat(web): utilize new tokenization-transition methodology
...
Note that we still need to keep the previous tokenization method's alignment style around for now - it's referenced by quite a number of unit tests for suggestions and reversions, which are currently written to leverage that. Updating the related methods will be its own unit of work.
2025-10-03 08:28:07 -05:00
Joshua Horton
2cadedef6d
fix(web): fix base context-state validation
...
After changes added in #14880 , one of our unit tests failed. After lots of investigation, it turns out that the `isSubstitutionAlignable` does not catch certain cases that `determineContextSlideTransform` does. The latter does need some improvement to work as a fix, but a small change lets it pick up the slack and replace the former.
As it's also used during context-state transition, re-using the same method in both places gives us far more confidence that we won't have an error the second time around.
Build-bot: skip build:web
Test-bot: skip
2025-10-02 16:29:03 -05:00
Joshua Horton
b48bf4334d
feat(web): adds legacy keyer that matches existing transform-tokenization behavior
...
Build-bot: skip test:web
Test-bot: skip
The new transform-tokenization "similarity" check added in #14822 is more limiting when only one tokenization is supported. As an interim measure, this commit and PR are designed to be reverted once the engine can handle multiple tokenizations properly and thus can swap fully to use of the new keying strategy.
2025-10-02 11:40:39 -05:00
Joshua Horton
7ca6f27962
feat(web): add method that evaluates precomputed tokenizations
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:40:09 -05:00
Joshua Horton
b7a1f8d73b
feat(web): compute number of tokens deleted in tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:34:27 -05:00
Joshua Horton
f9b80db5e3
fix(web): do not ignore empty context-tail tokens when computing tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:42 -05:00
Joshua Horton
95bda69e36
feat(web): add token-merge non-BMP string unit test
2025-10-02 11:33:03 -05:00
Joshua Horton
153415dd90
feat(web): add method for merging multiple ContextTokens into one
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:03 -05:00
Joshua Horton
050edaca21
fix(web): improve token-split handling of non-BMP text
2025-10-02 11:32:07 -05:00
Joshua Horton
ace4fefc0f
feat(web): add class method for splitting ContextToken instances
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:32:07 -05:00
Joshua Horton
9f5928eea0
change(web): tokenizedInputs -> inputs
2025-10-02 11:30:57 -05:00
Joshua Horton
fed69df41a
feat(web): build subsets of prior-state + input pairings with compatible tokenization results
...
Build-bot: skip build:web
Test-bot: skip
2025-09-29 15:01:48 -05:00
Joshua Horton
494fbe20b7
feat(web): define keying function that matches paths that result in compatible context tokenizations
...
This method is designed to identify changes in tokenization upon input that result in the same tokenization effects when applied. Such cases are safe to batch together when building search path extensions.
Build-bot: skip build:web
Test-bot: skip
2025-09-29 14:48:46 -05:00
Joshua Horton
4c8229707a
feat(web): track source text for edit-boundary source token
...
Build-bot: skip build:web
Test-bot: skip
2025-09-29 14:37:20 -05:00
Joshua Horton
e1e5ae24c5
change(web): attempt to improve nomenclature
2025-09-29 14:26:43 -05:00
Joshua Horton
8a6f4437cd
fix(web): implement proper non-BMP handling within precomputeTokenizationAfterInput
2025-09-29 11:37:57 -05:00
Joshua Horton
c7e5559d1a
Merge branch 'feat/web/tokenized-transform-assembly' into feat/web/boundary-shifting-transform-tokenization
2025-09-29 11:09:28 -05:00
Joshua Horton
f1ce917dc2
Merge branch 'feat/web/compute-merge-split-effects' into feat/web/tokenized-transform-assembly
2025-09-29 10:33:10 -05:00
Joshua Horton
978f9678a6
Merge branch 'feat/web/trace-insert-edits' into feat/web/compute-merge-split-effects
2025-09-29 10:32:00 -05:00
Joshua Horton
c477221e93
fix(web): implement traceInsertEdits non-BMP string handling
2025-09-29 10:31:30 -05:00
Joshua Horton
2e16e397d7
Merge branch 'feat/web/backward-slide-tokenization' into feat/web/trace-insert-edits
2025-09-29 10:29:55 -05:00
Joshua Horton
10ccc954ef
feat(web): add non-BMP test for edge-window-slide suite
2025-09-29 10:27:25 -05:00
Joshua Horton
cf017acb1c
Merge branch 'feat/web/build-token-aligned-edge-windows' into feat/web/backward-slide-tokenization
2025-09-29 10:23:50 -05:00
Joshua Horton
1ffed0991a
fix(web): fix edge-window construction handling of non-BMP chars
2025-09-29 10:22:58 -05:00
Joshua Horton
6cdba4297b
fix(web): patch up unit test
2025-09-25 15:28:59 -05:00
Joshua Horton
b17dc93f0d
Merge branch 'feat/web/tokenized-transform-assembly' into feat/web/boundary-shifting-transform-tokenization
2025-09-25 15:22:02 -05:00
Joshua Horton
04abb5e2fc
Merge branch 'feat/web/compute-merge-split-effects' into feat/web/tokenized-transform-assembly
2025-09-25 15:18:51 -05:00
Joshua Horton
b1a9e07ef3
feat(web): provide startOffset for detected splits based on start index within original token
2025-09-25 15:18:30 -05:00
Joshua Horton
686ce86082
fix(web): fix extended merges, splits
2025-09-25 15:17:37 -05:00
Joshua Horton
5bd7edbb28
Merge branch 'feat/web/trace-insert-edits' into feat/web/compute-merge-split-effects
2025-09-25 15:07:46 -05:00
Joshua Horton
d32a29f854
Merge branch 'feat/web/backward-slide-tokenization' into feat/web/trace-insert-edits
2025-09-25 15:07:03 -05:00
Joshua Horton
b7bc34d528
Merge branch 'feat/web/build-token-aligned-edge-windows' into feat/web/backward-slide-tokenization
2025-09-25 15:04:43 -05:00
Joshua Horton
414c6f12ab
Merge branch 'feat/web/context-slide-deltas' into feat/web/build-token-aligned-edge-windows
2025-09-25 15:03:40 -05:00
Joshua Horton
ec76f0dd5f
Merge branch 'refactor/web/source-input-vs-example-input' into feat/web/context-slide-deltas
2025-09-25 15:01:06 -05:00
Joshua Horton
6eedb2e7ab
change(web): revert no-longer-valid change
2025-09-25 14:59:49 -05:00
Joshua Horton
5dc28b5492
refactor(web): track original ContextToken source properties + reform use of .exampleText
2025-09-25 14:51:57 -05:00
Joshua Horton
2b64a0857b
docs(web): fix sourceText doc comment
2025-09-24 10:37:12 -05:00
Joshua Horton
602c74c6e5
feat(web): method now computes new tokenization pattern for original context
2025-09-22 12:26:10 -05:00
Joshua Horton
91f701b59f
feat(web): support transform tokenization even when existing boundaries shift
...
Related-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 12:26:05 -05:00
Joshua Horton
97e51a5a5d
change(web): construct alternate helper for assembly of tokenized transforms
...
This adds another helper designed to facilitate #14790 and is targeted to eventually replace our current `tokenizeTransform` method. We're not quite ready to drop `tokenizeTransform` and its helpers yet, though.
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:37:23 -05:00
Joshua Horton
7362d36066
feat(web): add helper for aligning tokens after merges, splits
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:36:12 -05:00
Joshua Horton
9ffc093ac4
feat(web): add helper for tokenizing Transform.insert via tokenization of applied context
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:34:40 -05:00
Joshua Horton
b1d3899c55
feat(web): adds helper method applyContextSlide for updating context tokenization before applying inputs
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:31:54 -05:00
Joshua Horton
1b39e92c16
change(web): edgeSliceIndex => sliceIndex
2025-09-22 09:30:17 -05:00
