When normalizing, we need to stop processing on an NFC boundary, not an
NFD boundary, to support normalizations such as in Bengali, where
appending `U+09D7` to a context of `U+0995 U+09C7` should result in
`U+0995 U+09CC`.
The specification is unclear on this; see https://unicode-org.atlassian.net/browse/CLDR-19218
This also updates the ldml keyboard unit test suite to support running
in full NFC mode (used in all Engine implementations) as well retaining
the NFD mode (now only used by the debugger).
Side note: the Bengali normalization failure case was picked up by the
improvements to the unit test suite, proving once again that good tests
are so valuable.
Fixes: #15491Fixes: #15505
Follows: #15488
Cherry-pick-of: #15506
Relates-to: CLDR-19218
Ensure that when a single backspace decomposes the last NFC character in
the app context, the remainder of the 'cluster' is preserved, matching
the implication of the CLDR keyboard specification.
This addresses the behavior in #15487 where the cached context became
out of sync with the app context after deleting an entire NFC cluster
such as ê, which caused a loop ending up with the entire context being
deleted, at which point the loop exited with a fail-safe.
Note that the LDML keyboard tests (ldml.cpp) do not currently exercise
the normalization code; this is a gap that should be addressed to ensure
that we are testing final application behavior.
This cherry-pick only adds the direct normalization patch and unit test
and skips the additional changes for logging.
Fixes: #15487
Cherry-pick-of: #15488
Two separate bugs addressed, with `index()` references and with
`context()` references -- both have the same root cause, of not taking
`nul` at the start of the context into account (as `nul` is not included
in the `m_miniContext` member, being a non-character). We already fixed
this issue for `if()` quite a long time ago, and some of the same
patterns can be with `m_miniContextIfLen` for example.
Fixes: #13304Fixes: #13316
- yes, expand 'other' to all possible combinations
- use ALT and CTRL instead of RALT,LALT and RCTRL,LCTRL in the key list (reduce expansions up to 4x)
Fixes: #12298
This change disables logging at compile time to work around #12661.
Logging can be enabled in the debugger, or by re-compiling with
`g_debug_KeymanLog` set to TRUE.
Related: #12661
Cherry-pick: #12674
- split keyboard loading into loading KMX file into blob and then
loading the keyboard processor from the blob.
- deprecate `km_core_keyboard_load`
- move file access next to deprecated method. This is now the only place
that loads a file in Core; unit tests have some more places that
load files.
- introduce GTest and add unit tests for loading from blob
Cherry-picked from `epic/web-core` branch.
Cherry-Pick-Commit: 1deaa323ad
Cherry-Pick-Commit: 59019cc8b7
Cherry-Pick-Commit: bc46458368
Cherry-Pick-Commit: d06aa29956
Cherry-Pick-Commit: 1c88166f6e
Cherry-Pick-Commit: 069cd21ecd
Cherry-Pick-Commit: 052ae2ec35
Cherry-Pick-Commit: 11a2a3ba3a
Part-of: #11293
Part-of: #8093
- move kmxplus processing into the base LdmlTestSource class
- add a function to traverse the layer list looking for keys to add
- The @@keylist keyword only has one example from each modifier set
Fixes: #12298
While the modifier state property in core's API is 16-bit, internally
ldml_processor supports the modifier flag LDML_KEYS_MOD_OTHER with a
value of `0x10000`, which requires widening the value (we match the
32-bit size of the KMX_DWORD value from KMX+).
Note: this is not yet well unit-tested.
Relates-to: #11072Fixes: #12057
* Remove unused parameters from SendDebugMessage functions
* Add SendDebugEntry and SendDebugExit functions for tracking
function entry/exit
* Add indenting and function names to log entries
* Remove unused debug functions
* Eliminate now-unused hwnd parameter in initialization functions
* Replace Log,LogEntry,LogExit functions with SendDebug equivalents in
kmtip
Many functions now have SendDebugEntry/SendDebugExit (or
return_SendDebugExit) pairs. It is important to SendDebugExit on all
returns from a function to keep the log indent depth consistent. In some
cases I chose not to add these logging calls, e.g. on frequently called
functions such as the message hooks.
- temporary header file generated by wasm during build
- built using icu
- test_unicode verifies the contents and synchronization with running ICU.
- this is used by util_normalize to provide normalization properties under wasm without needing to include ICU.
Fixes: #9467
- add core/tools build tree with custom targets
- add to core/build.sh to generate nfd_table.h
- test_unicode to validate Unicode version and compare NFD to actual ICU
- currently, linear search of the table.
- add a normalize_nfd() which takes a single codepoint
- temporarily keep ICU in actions_normalize.cpp and ldml_transforms.cpp
- expand wasm opts in unit tests
- always set to 0 for now (keep ICU around)
- set KMN_IN_LDML_TESTS in tests to keep ICU there for test and comparison
- add core_icu.cpp and put some utils there.
#9467
