Joshua Horton
50d9c1fb0f
Merge branch 'refactor/web/independent-compute-alignment' into change/web/map-transform-tokenization
2025-08-27 15:04:54 -05:00
Joshua Horton
fae2dec1fe
Merge branch 'fix/web/sliding-tokenization-alignment' into refactor/web/independent-compute-alignment
2025-08-27 15:04:29 -05:00
Joshua Horton
0937d0da4c
Merge branch 'fix/web/post-bksp-applies' into fix/web/sliding-tokenization-alignment
2025-08-26 08:54:45 -05:00
Joshua Horton
9362f2e0a0
fix(web): fix tokenization indices for niche suggestion-application cases
...
This also has the benefit of _greatly_ simplifying how we compute the tokenized transform indices. Kinda sad I didn't land on this approach the first time. (I actually felt like there _had_ to be something simpler... and it turns out there was!)
2025-08-26 08:35:55 -05:00
Joshua Horton
1f0fd8c8fe
Merge branch 'refactor/web/independent-compute-alignment' into change/web/map-transform-tokenization
2025-08-25 15:55:57 -05:00
Joshua Horton
1044d90e47
Merge branch 'fix/web/sliding-tokenization-alignment' into refactor/web/independent-compute-alignment
2025-08-25 15:55:29 -05:00
Joshua Horton
d35bd80a90
fix(web): fix last commit's fix - missed the return keyword a few times
2025-08-25 15:48:05 -05:00
Joshua Horton
e3510b9e51
Merge branch 'fix/web/sliding-tokenization-alignment' into refactor/web/independent-compute-alignment
2025-08-25 15:31:53 -05:00
Joshua Horton
4680295108
fix(web): fix error that occurs when applying suggestion to empty token
2025-08-25 15:22:45 -05:00
Joshua Horton
31e4a932d1
docs(web): Fix comment in alignment-helpers.ts
...
Co-authored-by: Darcy Wong <darcy_wong@sil.org>
2025-08-21 20:04:41 +07:00
Joshua Horton
1c28eeaf7c
docs(web): doc enhancements related to recent work in this PR
2025-08-20 12:44:36 -05:00
Joshua Horton
ef99dd49a9
change(web): map tokenized transform by index relative to context tail
...
With this change, the transform-tokenization methods now indicate which token (relative to the original context's tail token) each of the resulting tokenized transforms affects (and/or creates) in the output tokenization.
Fixes : #14361 .
Fixes : #14480 .
2025-08-20 12:21:13 -05:00
Joshua Horton
625f734223
refactor(web): move computeAlignment core to standalone method
...
Relates-to: #14480
This is being done to prepare for a fix to #14480 , which is needed to better handle complex input transforms and tokenization scenarios. The easiest way to consistently index the tokenized transforms... is to perform context alignment on the results of transform application.
2025-08-20 10:18:29 -05:00
Joshua Horton
e794f34dc9
Merge branch 'fix/web/post-bksp-applies' into fix/web/sliding-tokenization-alignment
2025-08-20 09:45:01 -05:00
Joshua Horton
9fbe0f4c40
fix(web): handle suggestion application after backspacing
2025-08-20 09:39:47 -05:00
Joshua Horton
dc230f68bc
fix(web): improve logic check for deleted token before attempting correction-search
2025-08-19 12:39:43 -05:00
Joshua Horton
9b28fb8e63
Merge branch 'change/web/cache-states-selectively' into fix/web/sliding-tokenization-alignment
2025-08-19 12:05:20 -05:00
Joshua Horton
fc0348585c
Merge branch 'refactor/web/analyze-state-transition' into change/web/cache-states-selectively
2025-08-19 12:04:54 -05:00
Joshua Horton
0b9d1018e5
Merge branch 'refactor/web/tokenization-transition' into refactor/web/analyze-state-transition
2025-08-19 12:00:21 -05:00
Joshua Horton
cc5a5fc38d
Merge branch 'change/web/suggestion-tracking' into refactor/web/tokenization-transition
2025-08-19 11:58:08 -05:00
Joshua Horton
08670c2e68
Merge branch 'change/web/cache-context-transitions-alt' into change/web/suggestion-tracking-alt
2025-08-18 13:56:24 -05:00
Joshua Horton
d6d44c4d1d
change(web): remove .get, force key type to number, base rewind on key value
2025-08-18 13:50:49 -05:00
Joshua Horton
470bf6a54f
Merge branch 'change/web/cache-states-selectively' into fix/web/sliding-tokenization-alignment
2025-08-15 14:06:09 -05:00
Joshua Horton
55b6b8bc5d
Merge branch 'refactor/web/analyze-state-transition' into change/web/cache-states-selectively
2025-08-15 14:05:42 -05:00
Joshua Horton
26444e7291
fix(web): fix empty context handling
2025-08-15 14:04:52 -05:00
Joshua Horton
c5892ab843
change(web): trim cached tokens, token data sliding out of context window
2025-08-15 11:21:51 -05:00
Joshua Horton
7515124c41
fix(web): prevent preserved leading-token text from breaking alignment
...
Prevents scenarios in which a word sliding out of the predictive-text engine's available context window causes the engine's context alignment to fail.
Our cached tokenization data preserves all text for tokens, even those partially slid out of the active context window. We could either adjust/replace the token or allow the leading token to align with notably decreased restrictions. ... or both, to be extra robust (in case of large transform input).
This commit allows alignment with notably decreased restrictions.
2025-08-15 11:21:49 -05:00
Joshua Horton
3e3e5713e1
change(web): do not cache context states that will no longer be referenced
2025-08-15 11:14:40 -05:00
Joshua Horton
01389555d2
docs(web): more minor doc polishing
2025-08-15 11:12:33 -05:00
Joshua Horton
0562eca3e0
docs(web): improve file header
2025-08-15 11:11:31 -05:00
Joshua Horton
8f920c4b51
docs(web): improve analyzeTransition documentation
2025-08-15 11:10:39 -05:00
Joshua Horton
6a6c9e4923
refactor(web): move attemptMatchContext to ContextState.analyzeTransition
2025-08-15 10:44:42 -05:00
Joshua Horton
b73f6d3a52
fix(web): prevent post-application suggestions from overwriting applied suggestion set
2025-08-15 10:28:44 -05:00
Joshua Horton
54c19e36cb
refactor(web): refactor aligned tokenization-transition operations into separate method
...
I've really been looking forward to figuring this one out. This refactor spins off the bulk of `attemptMatchContext` into a _class_ method on `ContextTokenization` - `.transitionTo` - which uses alignment data to build the new tokenization for incoming context based on the old tokenization in a clear manner.
By factorizing the method in this manner, we pave the way to allow this operation to be performed for different tokenization patterns based on either the initial or the final context (or both!) in the future - something epic-dict-breaker models will likely need to leverage. It also makes the roles of the two halves of the code _far_ clearer than it was before, in my opinion.
2025-08-15 10:28:44 -05:00
Joshua Horton
b200f19d46
refactor(web): hoist applySuggestion refactor from child branch
2025-08-15 10:21:26 -05:00
Joshua Horton
eebf85a888
docs(web): documents parameter added to computeAlignment
2025-08-15 10:21:26 -05:00
Joshua Horton
5804bc3586
change(web): copy applied-transition ID when cloning tokens
2025-08-15 10:21:26 -05:00
Joshua Horton
3cc19c22d7
feat(web): adds ContextTransition unit tests
2025-08-15 10:21:26 -05:00
Joshua Horton
bbb7dc68a4
change(web): annotate tokens affected by applied suggestions
2025-08-15 10:21:26 -05:00
Joshua Horton
ca1205ad39
refactor(web): refactor reconstruction of transitions for reverted suggestions
2025-08-15 10:21:26 -05:00
Joshua Horton
79b33c4f1c
fix(web): add special handling for alignment when applying suggestions
2025-08-15 10:21:26 -05:00
Joshua Horton
c67ea99484
change(web): rework applied-suggestion tracking + reversions
2025-08-15 10:21:26 -05:00
Joshua Horton
dc9276979e
change(web): cache context transition objects, not just context state
2025-08-14 14:22:25 -05:00
Joshua Horton
f224b8bfa5
change(web): add standard header, reorder imports
2025-08-14 14:09:59 -05:00
Joshua Horton
36defc42f3
Merge branch 'refactor/web/distribution-tokenization' into refactor/web/pred-context-transition
2025-08-14 14:08:05 -05:00
Joshua Horton
3fe1b17239
Merge branch 'refactor/web/init-state-from-reset' into refactor/web/distribution-tokenization
2025-08-14 13:55:46 -05:00
Joshua Horton
8a1ae2d460
Merge branch 'refactor/web/pred-context-state' into refactor/web/init-state-from-reset
2025-08-14 13:38:52 -05:00
Joshua Horton
d3fabb4848
docs(web): adds standard header to new file, reorders imports
2025-08-14 13:32:20 -05:00
Joshua Horton
a4ac1a2f4d
Merge branch 'refactor/web/tokenization-alignment' into refactor/web/pred-context-state
2025-08-14 13:28:27 -05:00
Joshua Horton
c153dc73c4
Merge branch 'refactor/web/context-alignment-helpers' into refactor/web/tokenization-alignment
2025-08-14 13:19:21 -05:00
