Joshua Horton
41619fed1f
Merge branch 'feat/web/evaluate-precomputed-tokenization' into feat/web/interim-legacy-keyer
2025-10-08 11:41:39 -05:00
Joshua Horton
8dd5748fec
docs(web): "assumption", not "assertion"
2025-10-08 11:37:36 -05:00
Joshua Horton
4698f22c86
Merge branch 'feat/web/compute-removed-token-count' into feat/web/evaluate-precomputed-tokenization
2025-10-08 08:52:04 -05:00
Joshua Horton
b2f460d0be
Merge branch 'fix/web/empty-context-tail-handling' into feat/web/compute-removed-token-count
2025-10-08 08:51:22 -05:00
Joshua Horton
30f0eee46b
Merge branch 'feat/web/merge-context-tokens' into fix/web/empty-context-tail-handling
2025-10-08 08:50:08 -05:00
Joshua Horton
886642c67d
Merge branch 'feat/web/split-context-tokens' into feat/web/merge-context-tokens
2025-10-08 08:49:25 -05:00
Joshua Horton
2b290955a4
Merge branch 'feat/web/tokenization-subset-builder' into feat/web/split-context-tokens
2025-10-08 08:45:19 -05:00
Joshua Horton
8a69cd4689
Merge branch 'feat/web/key-common-result-tokenizations' into feat/web/tokenization-subset-builder
2025-10-08 08:43:28 -05:00
Joshua Horton
9c8747d9c8
Merge branch 'feat/web/track-boundary-token-source' into feat/web/key-common-result-tokenizations
2025-10-08 08:42:01 -05:00
Joshua Horton
164b6306ef
Merge branch 'feat/web/boundary-shifting-transform-tokenization' into feat/web/track-boundary-token-source
2025-10-08 08:40:55 -05:00
Joshua Horton
01a0eeec19
change(web): renames precomputeTokenizationAfterInput to mapWhitespacedTokenization
2025-10-08 08:40:15 -05:00
Joshua Horton
b48bf4334d
feat(web): adds legacy keyer that matches existing transform-tokenization behavior
...
Build-bot: skip test:web
Test-bot: skip
The new transform-tokenization "similarity" check added in #14822 is more limiting when only one tokenization is supported. As an interim measure, this commit and PR are designed to be reverted once the engine can handle multiple tokenizations properly and thus can swap fully to use of the new keying strategy.
2025-10-02 11:40:39 -05:00
Joshua Horton
7ca6f27962
feat(web): add method that evaluates precomputed tokenizations
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:40:09 -05:00
Joshua Horton
b7a1f8d73b
feat(web): compute number of tokens deleted in tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:34:27 -05:00
Joshua Horton
f9b80db5e3
fix(web): do not ignore empty context-tail tokens when computing tokenization transition
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:42 -05:00
Joshua Horton
153415dd90
feat(web): add method for merging multiple ContextTokens into one
...
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:33:03 -05:00
Joshua Horton
050edaca21
fix(web): improve token-split handling of non-BMP text
2025-10-02 11:32:07 -05:00
Joshua Horton
ace4fefc0f
feat(web): add class method for splitting ContextToken instances
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-10-02 11:32:07 -05:00
Joshua Horton
9f5928eea0
change(web): tokenizedInputs -> inputs
2025-10-02 11:30:57 -05:00
Joshua Horton
f1b8cb2457
docs(web): assertion -> assumption
2025-09-30 09:18:56 -05:00
Joshua Horton
fed69df41a
feat(web): build subsets of prior-state + input pairings with compatible tokenization results
...
Build-bot: skip build:web
Test-bot: skip
2025-09-29 15:01:48 -05:00
Joshua Horton
494fbe20b7
feat(web): define keying function that matches paths that result in compatible context tokenizations
...
This method is designed to identify changes in tokenization upon input that result in the same tokenization effects when applied. Such cases are safe to batch together when building search path extensions.
Build-bot: skip build:web
Test-bot: skip
2025-09-29 14:48:46 -05:00
Joshua Horton
4c8229707a
feat(web): track source text for edit-boundary source token
...
Build-bot: skip build:web
Test-bot: skip
2025-09-29 14:37:20 -05:00
Joshua Horton
e1e5ae24c5
change(web): attempt to improve nomenclature
2025-09-29 14:26:43 -05:00
Joshua Horton
8a6f4437cd
fix(web): implement proper non-BMP handling within precomputeTokenizationAfterInput
2025-09-29 11:37:57 -05:00
Joshua Horton
c7e5559d1a
Merge branch 'feat/web/tokenized-transform-assembly' into feat/web/boundary-shifting-transform-tokenization
2025-09-29 11:09:28 -05:00
Joshua Horton
f1ce917dc2
Merge branch 'feat/web/compute-merge-split-effects' into feat/web/tokenized-transform-assembly
2025-09-29 10:33:10 -05:00
Joshua Horton
978f9678a6
Merge branch 'feat/web/trace-insert-edits' into feat/web/compute-merge-split-effects
2025-09-29 10:32:00 -05:00
Joshua Horton
c477221e93
fix(web): implement traceInsertEdits non-BMP string handling
2025-09-29 10:31:30 -05:00
Joshua Horton
2e16e397d7
Merge branch 'feat/web/backward-slide-tokenization' into feat/web/trace-insert-edits
2025-09-29 10:29:55 -05:00
Joshua Horton
cf017acb1c
Merge branch 'feat/web/build-token-aligned-edge-windows' into feat/web/backward-slide-tokenization
2025-09-29 10:23:50 -05:00
Joshua Horton
1ffed0991a
fix(web): fix edge-window construction handling of non-BMP chars
2025-09-29 10:22:58 -05:00
Joshua Horton
b17dc93f0d
Merge branch 'feat/web/tokenized-transform-assembly' into feat/web/boundary-shifting-transform-tokenization
2025-09-25 15:22:02 -05:00
Joshua Horton
04abb5e2fc
Merge branch 'feat/web/compute-merge-split-effects' into feat/web/tokenized-transform-assembly
2025-09-25 15:18:51 -05:00
Joshua Horton
b1a9e07ef3
feat(web): provide startOffset for detected splits based on start index within original token
2025-09-25 15:18:30 -05:00
Joshua Horton
686ce86082
fix(web): fix extended merges, splits
2025-09-25 15:17:37 -05:00
Joshua Horton
5bd7edbb28
Merge branch 'feat/web/trace-insert-edits' into feat/web/compute-merge-split-effects
2025-09-25 15:07:46 -05:00
Joshua Horton
d32a29f854
Merge branch 'feat/web/backward-slide-tokenization' into feat/web/trace-insert-edits
2025-09-25 15:07:03 -05:00
Joshua Horton
b7bc34d528
Merge branch 'feat/web/build-token-aligned-edge-windows' into feat/web/backward-slide-tokenization
2025-09-25 15:04:43 -05:00
Joshua Horton
414c6f12ab
Merge branch 'feat/web/context-slide-deltas' into feat/web/build-token-aligned-edge-windows
2025-09-25 15:03:40 -05:00
Joshua Horton
ec76f0dd5f
Merge branch 'refactor/web/source-input-vs-example-input' into feat/web/context-slide-deltas
2025-09-25 15:01:06 -05:00
Joshua Horton
6eedb2e7ab
change(web): revert no-longer-valid change
2025-09-25 14:59:49 -05:00
Joshua Horton
5dc28b5492
refactor(web): track original ContextToken source properties + reform use of .exampleText
2025-09-25 14:51:57 -05:00
Joshua Horton
2b64a0857b
docs(web): fix sourceText doc comment
2025-09-24 10:37:12 -05:00
Joshua Horton
602c74c6e5
feat(web): method now computes new tokenization pattern for original context
2025-09-22 12:26:10 -05:00
Joshua Horton
91f701b59f
feat(web): support transform tokenization even when existing boundaries shift
...
Related-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 12:26:05 -05:00
Joshua Horton
97e51a5a5d
change(web): construct alternate helper for assembly of tokenized transforms
...
This adds another helper designed to facilitate #14790 and is targeted to eventually replace our current `tokenizeTransform` method. We're not quite ready to drop `tokenizeTransform` and its helpers yet, though.
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:37:23 -05:00
Joshua Horton
7362d36066
feat(web): add helper for aligning tokens after merges, splits
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:36:12 -05:00
Joshua Horton
9ffc093ac4
feat(web): add helper for tokenizing Transform.insert via tokenization of applied context
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:34:40 -05:00
Joshua Horton
b1d3899c55
feat(web): adds helper method applyContextSlide for updating context tokenization before applying inputs
...
Relates-to: #14679
Build-bot: skip build:web
Test-bot: skip
2025-09-22 09:31:54 -05:00
