Commit graph

634 commits

Author SHA1 Message Date
Marc Durdin
5790f634cf
Merge pull request #11101 from keymanapp/fix/core/11067-fix-actions-normalize-utf32
fix(core): fix pointer math in actions_normalize() 🙀
2024-03-28 19:18:15 +11:00
Steven R. Loomis
92a4a2a0a2
Merge pull request #11094 from keymanapp/chore/core/11057-optimize-marker-context
chore(core): optimize ldml_event_state::emit_difference() when no diff 🙀
2024-03-28 00:13:45 -05:00
Steven R. Loomis
7106b662b7
Merge pull request #11100 from keymanapp/fix/core/11067-fix-actions-normalize-output-string
fix(core): actions_normalize() length and dead store fix 🙀
2024-03-28 00:13:21 -05:00
Steven R. Loomis
adc421974c chore(core): fix actions_normalize() pointer math - tests
- fix comments yet again
- add regression test in test_actions_normalize.cpp

For: #11067
2024-03-27 23:55:01 -05:00
Steven R. Loomis
d894db78f6 fix(core): add an assert on code_points_to_delete 2024-03-27 23:25:43 -05:00
Steven R. Loomis
ca34b5f00f fix(core): fix actions_normalize() UChar32 calculation
- UnicodeString is UTF-16, but can be used with UTF-32 boundaries.
- fix calculation of code_points_to_delete

Fixes: #11067
2024-03-27 19:37:08 -05:00
Steven R. Loomis
3f42b0b831 fix(core): fix actions_normalize() use of UnicodeString
- output[0] was used as a boolean, but needed to check output.isEmpty() instead
- dead store to  actions.code_points_to_delete

For: #11067 parts 1 and 2
2024-03-27 19:13:01 -05:00
Steven R. Loomis
8b5723028c chore(core): optimize ldml_event_state::emit_difference() when no difference
#11057
2024-03-27 08:18:03 -05:00
Steven R. Loomis
e8a8590eee fix(core): revert some interim fixes
- ldml_test_source fix moved off to issue #11083
- an optimization in ldml_processor also rolled back
2024-03-27 08:05:39 -05:00
Steven R. Loomis
18c6ecfc08 fix(core): fix 2 marker cases in ldml_event_state::emit_difference 🙀
- handle the case where old and new context strings are the same (i.e. no work to do).
- fix pointer arithmetic error in #10356 - the special case where one marker is being replaced by another. The named regression test didn't actually hit this case.

Fixes: bug(core): 'string too long' #11057
2024-03-25 16:45:01 -05:00
Steven R. Loomis
17a653fa19 chore(core): ldml update out of techpreview 🙀
- updates to support techpreview -> 45
- test changes

Fixes: #10900
2024-03-02 22:40:39 -06:00
Steven R. Loomis
2ee402c2b8
Update core/src/ldml/ldml_transforms.cpp
Co-authored-by: Marc Durdin <marc@durdin.net>
2024-02-19 12:37:26 -06:00
Steven R. Loomis
6b229ffa14 fix(core): fix output append logic in any_group::apply_transform
- logic updating intermediate 'output' was incorrect
- update comments
- also replace size() with length() several places

Fixes #10736
2024-02-17 23:50:55 -06:00
Steven R. Loomis
ceb990e713 chore(core): more shuffling
For #10736
2024-02-16 18:27:35 -06:00
Steven R. Loomis
746af68afa chore(core): shuffle transform matcher. re-enable failing case
For #10736
2024-02-16 12:49:19 -06:00
Steven R. Loomis
e7b31113a3 chore(core): remove obsolete TODOs. Temporary change to test.
- TODO: 10734 There's a bug in the matching still. Come back and fix this, but the crasher is fixed.

For #10734
2024-02-15 16:21:53 -06:00
Steven R. Loomis
5edb70f07a fix(core): fix for ldml_transforms.cpp
For #10734
2024-02-15 15:26:30 -06:00
Marc Durdin
64daa6182b
Merge pull request #10618 from keymanapp/fix/core/10615-10616-10617-core-surrogates-and-app-context-markers-and-memory-leak
fix(core): surrogate handling, markers in app context, and memory leak
2024-02-07 14:43:12 +11:00
Marc Durdin
bcc18bfe8d
Merge pull request #10607 from keymanapp/fix/core/10605-app-context-copy-strip-markers
fix(core): strip markers in `actions_update_app_context_nfu()`
2024-02-07 14:43:01 +11:00
Marc Durdin
6c4d5ff7fa
Merge pull request #10585 from keymanapp/fix/core/10582-km_core_state_get_actions-idempotency
fix(core): make `km_core_state_get_actions()` idempotent
2024-02-07 14:42:50 +11:00
Steven R. Loomis
889e9c1581
Merge pull request #10586 from keymanapp/feat/developer/10554-sometimes-no-norm 2024-02-06 21:38:39 -06:00
Marc Durdin
0cc6901085
Merge branch 'fix/core/10605-app-context-copy-strip-markers' into fix/core/10615-10616-10617-core-surrogates-and-app-context-markers-and-memory-leak 2024-02-07 12:33:33 +11:00
Marc Durdin
232df37400
chore: Update core/src/state.cpp 2024-02-07 12:29:45 +11:00
Marc Durdin
c807415563
chore: Apply suggestions from code review
Co-authored-by: Steven R. Loomis <srl295@gmail.com>
2024-02-07 12:20:56 +11:00
Steven R. Loomis
15b7a42829 feat(core): support normalization=disabled 🙀
- clean up process_output:  remove some unnecessary steps,
refactor into 3 functions, clarify
- reorder functions for ldml_state
- remove extraneous assert() clutter when the normalization
functions already assert/DebugLog at the innermost level

#10554
2024-02-06 16:09:18 -06:00
Marc Durdin
c9eb545c05 refactor(core): more renames to satisfy compiler on linux
Shadowed types are not permitted.
2024-02-05 12:20:44 +07:00
Marc Durdin
6d8cae1f2e refactor(core): is_context_changed() into two functions
Refactors `is_context_changed()` into two functions,
`get_context_change()` and `get_context_items_change()`, to eliminate
use of `goto` for cleanup, and more accurately reflect the purpose of
the function. Also renames some variables referencing the return value
from these functions.

These functions would probably benefit from unit tests proving them.
2024-02-05 12:07:41 +07:00
Marc Durdin
fe8eba176c
Merge branch 'fix/core/10605-app-context-copy-strip-markers' into fix/core/10615-10616-10617-core-surrogates-and-app-context-markers-and-memory-leak 2024-02-05 13:45:03 +11:00
Marc Durdin
49eb6f42a9
Merge branch 'fix/core/10582-km_core_state_get_actions-idempotency' into fix/core/10605-app-context-copy-strip-markers 2024-02-05 13:44:56 +11:00
Marc Durdin
70ba1d1c1e
Merge branch 'chore/core/10520-hide-old-action-apis' into fix/core/10582-km_core_state_get_actions-idempotency 2024-02-05 13:44:47 +11:00
Marc Durdin
aa80992fb8
Merge branch 'feat/windows/use-km-core-actions-struct' into chore/core/10520-hide-old-action-apis 2024-02-05 13:44:36 +11:00
Marc Durdin
9ba919c2a9 fix(core): surrogate handling, markers in app context, and memory leak
Fixes #10615 (surrogate handling, km_core_state_context_set_if_needed).
Fixes #10616 (markers in app context, in unit tests)
Fixes #10617 (memory leak in replace_context).

The surrogate handling and markers in app context bugs were quite
tangled, so opted to fix those both together in one commit. The memory
leak was spotted while verifying the changes for surrogate handling.

Surrogates were not handled in is_context_unchanged, and opted to
refactor to work directly with km_core_context_item arrays rather than
continue with the string-based comparison, for simplicity.

While writing the corresponding unit tests, realised that the
app_context checks were incorrect in a number of existing tests, and so
fixed up the existing tests at the same time as writing up the new
tests. The only changes to core for this were to add assertions for the
app_context to ensure that markers are not inadvertently added.
2024-02-03 09:16:16 +07:00
Marc Durdin
9e50f50687 fix(core): strip markers in actions_update_app_context_nfu()
Fixes #10605.
2024-02-02 15:10:05 +07:00
Steven R. Loomis
71443836a2 feat(developer): support normalization=disabled 🙀
- avoid dead store to transforms class

#10554
2024-02-01 19:05:31 -06:00
Steven R. Loomis
7572c9476e feat(developer): support normalization=disabled 🙀
- main core changes and test changes

#10554
2024-02-01 17:32:42 -06:00
Marc Durdin
aad7c0a42b chore(core): fixup typos 2024-02-01 15:05:24 +07:00
Marc Durdin
0b7556870f
chore: Apply suggestions from code review 2024-02-01 18:56:26 +11:00
Marc Durdin
15f2412717 chore(core): build whackamole 2024-02-01 12:50:23 +07:00
Marc Durdin
2abaead178 chore(core): string include 2024-02-01 12:38:58 +07:00
Marc Durdin
56cda3fff4 chore(core): address cross-platform builds 2024-02-01 12:17:10 +07:00
Marc Durdin
1b88ab99be fix(core): make km_core_state_get_actions() idempotent
Fixes #10582.

Returned struct from the `km_core_state_get_actions()` API is now owned
by the state object. This is a change in API contract. The corresponding
`km_core_actions_dispose()` API has been made private, because there is
never any need for API consumers to call it. As this change is happening
in alpha, we won't bump the ABI version.

`state->app_context` is now guaranteed to be in sync with
`state->context` after `km_core_process_event()`, with the actions
struct populated at that point.

The next and final step in this refactor is to remove the actions queue
altogether from the Core and make that a kmn-only concept, but that will
wait until 18.0. The only consumers of the actions queue are the
kmn-specific interactive debugger in Keyman Developer, and kmn-specific
Input Method eXtensions in Keyman Engine for Windows.
2024-02-01 11:28:08 +07:00
Steven R. Loomis
a28cee0fac
Merge branch 'feat/core/10317-double-marker2-epic-ldml' into feat/core/10317-simplify-marker-epic-ldml 2024-01-31 22:25:45 -06:00
Steven R. Loomis
006b955f8a
Merge branch 'feat/core/10317-double-marker-epic-ldml' into feat/core/10317-double-marker2-epic-ldml 2024-01-31 22:24:46 -06:00
Steven R. Loomis
ec03a681f7 feat(core): reorder processing 🙀
- fix quickcheck of add_back_markers()

For: #10516
2024-01-31 22:16:50 -06:00
Steven R. Loomis
78f81516c0 feat(core): report on (lack of) normalization 🙀
- report normalization

#10468
2024-01-31 18:22:23 -06:00
Steven R. Loomis
6c1447bb4b feat(core,developer): simplify markers 🙀
- clean up removing multi-segment processing
2024-01-31 18:17:17 -06:00
Steven R. Loomis
32d2e6ea2c feat(core,developer): simplify markers 🙀
- maybe don't need NFD table

#10317
2024-01-31 18:15:37 -06:00
Steven R. Loomis
600fdcc2c8 feat(core): double markers once again 🙀
- C++ side passes

Fixes: #10516
2024-01-31 18:05:53 -06:00
Steven R. Loomis
2891d19d88 feat(core): double markers once again 🙀
- C++ side

Fixes: #10516
2024-01-31 18:05:53 -06:00
Steven R. Loomis
2c92cfea4b feat(core): reorder processing 🙀
- update per code review
- move quickcheck of map into add_back_markers()

For: #10516
2024-01-31 17:44:21 -06:00