Joshua Horton
1ce8b7589a
Merge branch 'refactor/web/preservation-transforms' into refactor/web/delayed-reversion-via-tokenization-analysis
2025-10-16 10:18:50 -05:00
Joshua Horton
1942f8f8b1
Merge branch 'feat/web/link-new-tokenization-transitioner' into refactor/web/preservation-transforms
2025-10-16 10:18:06 -05:00
Joshua Horton
d376c777a4
Merge branch 'fix/web/base-context-state-validation' into feat/web/link-new-tokenization-transitioner
2025-10-16 10:17:32 -05:00
Joshua Horton
4612b8cd2a
Merge branch 'epic/autocorrect' into fix/web/base-context-state-validation
2025-10-16 09:54:55 -05:00
Joshua Horton
aaec2837ca
Merge pull request #14825 from keymanapp/feat/web/interim-legacy-keyer
...
feat(web): adds legacy keyer that matches existing transform-tokenization behavior 🚂
2025-10-16 21:49:54 +07:00
Joshua Horton
ebc264ac15
Merge branch 'epic/autocorrect' into feat/web/evaluate-precomputed-tokenization
2025-10-16 09:17:22 -05:00
Joshua Horton
66209db351
Merge pull request #14876 from keymanapp/feat/web/compute-removed-token-count
...
feat(web): compute number of tokens deleted in tokenization transition 🚂
2025-10-16 21:09:35 +07:00
Joshua Horton
1a78cc5a25
Merge pull request #14875 from keymanapp/fix/web/empty-context-tail-handling
...
fix(web): do not ignore empty context-tail tokens when computing tokenization transition 🚂
2025-10-16 21:09:26 +07:00
Keyman Server
0753cc548c
Merge branch 'epic/autocorrect' into chore/A19S13-merge-master-into-autocorrect
2025-10-10 05:04:36 +02:00
Joshua Horton
490c5e6751
Merge branch 'feat/web/interim-legacy-keyer' into fix/web/base-context-state-validation
2025-10-08 11:43:06 -05:00
Joshua Horton
41619fed1f
Merge branch 'feat/web/evaluate-precomputed-tokenization' into feat/web/interim-legacy-keyer
2025-10-08 11:41:39 -05:00
Joshua Horton
8dd5748fec
docs(web): "assumption", not "assertion"
2025-10-08 11:37:36 -05:00
Joshua Horton
b28bb2deb7
Merge pull request #14841 from keymanapp/feat/web/merge-context-tokens
...
feat(web): add method for merging multiple ContextTokens into one 🚂
2025-10-08 22:35:38 +07:00
Joshua Horton
da444878ff
Merge pull request #14840 from keymanapp/feat/web/split-context-tokens
...
feat(web): add class method for splitting ContextToken instances 🚂
2025-10-08 22:35:29 +07:00
Joshua Horton
d70d7a75c9
Merge pull request #14823 from keymanapp/feat/web/tokenization-subset-builder
...
feat(web): build subsets of prior-state + input pairings with compatible tokenization results 🚂
2025-10-08 22:35:20 +07:00
Joshua Horton
e95422565d
Merge pull request #14822 from keymanapp/feat/web/key-common-result-tokenizations
...
feat(web): define keying function that matches paths that result in compatible context tokenizations 🚂
2025-10-08 22:35:09 +07:00
Joshua Horton
05a89a07a4
Merge pull request #14820 from keymanapp/feat/web/track-boundary-token-source
...
feat(web): track source input ranges for the edit-boundary source token 🚂
2025-10-08 22:34:58 +07:00
Joshua Horton
eb3a1d86bb
Merge pull request #14790 from keymanapp/feat/web/boundary-shifting-transform-tokenization
...
feat(web): precompute transform tokenization while handling when existing boundaries shift 🚂
2025-10-08 22:34:48 +07:00
Joshua Horton
4698f22c86
Merge branch 'feat/web/compute-removed-token-count' into feat/web/evaluate-precomputed-tokenization
2025-10-08 08:52:04 -05:00
Joshua Horton
b2f460d0be
Merge branch 'fix/web/empty-context-tail-handling' into feat/web/compute-removed-token-count
2025-10-08 08:51:22 -05:00
Joshua Horton
30f0eee46b
Merge branch 'feat/web/merge-context-tokens' into fix/web/empty-context-tail-handling
2025-10-08 08:50:08 -05:00
Joshua Horton
886642c67d
Merge branch 'feat/web/split-context-tokens' into feat/web/merge-context-tokens
2025-10-08 08:49:25 -05:00
Joshua Horton
2b290955a4
Merge branch 'feat/web/tokenization-subset-builder' into feat/web/split-context-tokens
2025-10-08 08:45:19 -05:00
Joshua Horton
8a69cd4689
Merge branch 'feat/web/key-common-result-tokenizations' into feat/web/tokenization-subset-builder
2025-10-08 08:43:28 -05:00
Joshua Horton
9c8747d9c8
Merge branch 'feat/web/track-boundary-token-source' into feat/web/key-common-result-tokenizations
2025-10-08 08:42:01 -05:00
Joshua Horton
164b6306ef
Merge branch 'feat/web/boundary-shifting-transform-tokenization' into feat/web/track-boundary-token-source
2025-10-08 08:40:55 -05:00
Joshua Horton
01a0eeec19
change(web): renames precomputeTokenizationAfterInput to mapWhitespacedTokenization
2025-10-08 08:40:15 -05:00
Joshua Horton
604b2526a1
refactor(web): use tokenization analysis to maintain delayed reversion feature
2025-10-07 11:29:47 -05:00
Joshua Horton
b2cb13741a
refactor(web): rework preservation transforms to prep for multi-tokenization
...
Build-bot: skip build:web
Test-bot: skip
2025-10-07 11:25:13 -05:00
Joshua Horton
c3e5c66a6e
docs(web): thoughts toward upcoming changes for preservation transforms
2025-10-07 11:19:15 -05:00
Joshua Horton
39dd3b50a3
change(web): clean up computation of preservationTransform
2025-10-07 09:20:37 -05:00
Joshua Horton
199c962f6f
Merge pull request #14807 from keymanapp/feat/web/tokenized-transform-assembly
...
change(web): provide alternate helper for assembly of tokenized transforms 🚂
2025-10-06 21:52:03 +07:00
Joshua Horton
df2a868605
Merge pull request #14806 from keymanapp/feat/web/compute-merge-split-effects
...
feat(web): add helper for aligning tokens after merges, splits 🚂
2025-10-06 21:51:52 +07:00
Joshua Horton
e8f28fdf36
Merge pull request #14805 from keymanapp/feat/web/trace-insert-edits
...
feat(web): add helper for tokenizing Transform.insert via tokenization of applied context 🚂
2025-10-06 21:51:38 +07:00
Joshua Horton
a9a761443f
Merge pull request #14754 from keymanapp/feat/web/backward-slide-tokenization
...
feat(web): add helper method for updating context tokenization before applying inputs 🚂
2025-10-06 21:51:20 +07:00
Joshua Horton
6800734d39
Merge pull request #14800 from keymanapp/feat/web/build-token-aligned-edge-windows
...
feat(web): build token-aligned edge windows 🚂
2025-10-06 21:51:10 +07:00
Joshua Horton
e681dc46ed
Merge pull request #14751 from keymanapp/feat/web/context-slide-deltas
...
feat(web): add method to determine the change to context resulting from a context window slide 🚂
2025-10-06 21:50:59 +07:00
Joshua Horton
2db1e3fd20
Merge pull request #14756 from keymanapp/refactor/web/source-input-vs-example-input
...
refactor(web): differentiate between true and represented inputs for context tokens 🚂
2025-10-06 21:50:46 +07:00
Eberhard Beilharz
229bdf8270
maint(resources): nest unit tests under parent block
...
This adds a modified `mocha-teamcity-reporter` that allows to nest the
unit tests under the parent block.
Fixes : #14839
Test-bot: skip
2025-10-06 16:09:47 +02:00
Joshua Horton
0cf7930cad
change(web): remove old transitionTo method in favor of new tokenization-transition method
2025-10-03 08:28:07 -05:00
Joshua Horton
49af0c18a5
feat(web): utilize new tokenization-transition methodology
...
Note that we still need to keep the previous tokenization method's alignment style around for now - it's referenced by quite a number of unit tests for suggestions and reversions, which are currently written to leverage that. Updating the related methods will be its own unit of work.
2025-10-03 08:28:07 -05:00
Joshua Horton
2cadedef6d
fix(web): fix base context-state validation
...
After changes added in #14880 , one of our unit tests failed. After lots of investigation, it turns out that the `isSubstitutionAlignable` does not catch certain cases that `determineContextSlideTransform` does. The latter does need some improvement to work as a fix, but a small change lets it pick up the slack and replace the former.
As it's also used during context-state transition, re-using the same method in both places gives us far more confidence that we won't have an error the second time around.
Build-bot: skip build:web
Test-bot: skip
2025-10-02 16:29:03 -05:00
Joshua Horton
b48bf4334d
feat(web): adds legacy keyer that matches existing transform-tokenization behavior
...
Build-bot: skip test:web
Test-bot: skip
The new transform-tokenization "similarity" check added in #14822 is more limiting when only one tokenization is supported. As an interim measure, this commit and PR are designed to be reverted once the engine can handle multiple tokenizations properly and thus can swap fully to use of the new keying strategy.
2025-10-02 11:40:39 -05:00
Joshua Horton
7ca6f27962
feat(web): add method that evaluates precomputed tokenizations
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:40:09 -05:00
Joshua Horton
b7a1f8d73b
feat(web): compute number of tokens deleted in tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:34:27 -05:00
Joshua Horton
f9b80db5e3
fix(web): do not ignore empty context-tail tokens when computing tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:42 -05:00
Joshua Horton
153415dd90
feat(web): add method for merging multiple ContextTokens into one
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:03 -05:00
Joshua Horton
050edaca21
fix(web): improve token-split handling of non-BMP text
2025-10-02 11:32:07 -05:00
Joshua Horton
ace4fefc0f
feat(web): add class method for splitting ContextToken instances
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:32:07 -05:00
Joshua Horton
9f5928eea0
change(web): tokenizedInputs -> inputs
2025-10-02 11:30:57 -05:00
