4 commits

Author	SHA1	Message	Date
Marc Durdin	766c6992ed	fix(core): normalization segment should end on NFC boundary, not NFD ... When normalizing, we need to stop processing on an NFC boundary, not an NFD boundary, to support normalizations such as in Bengali, where appending `U+09D7` to a context of `U+0995 U+09C7` should result in `U+0995 U+09CC`. The specification is unclear on this; see https://unicode-org.atlassian.net/browse/CLDR-19218 This also updates the ldml keyboard unit test suite to support running in full NFC mode (used in all Engine implementations) as well retaining the NFD mode (now only used by the debugger). Side note: the Bengali normalization failure case was picked up by the improvements to the unit test suite, proving once again that good tests are so valuable. Fixes: #15491 Fixes: #15505 Follows: #15488 Relates-to: CLDR-19218	2026-01-28 14:32:25 +11:00
Steven R. Loomis	5d420248eb	feat(core): move more normalization logic into JS ... - major redo of actions_normalize - into UTF-32 and not using ICU directly - add some utilities: u32len, u32dup, context_items_from_utf32 #9467	2024-05-24 18:20:30 -05:00
Steven R. Loomis	a80a0a7dcb	feat(core): move more normalization logic into JS ... - add a normalize_nfd() which takes a single codepoint - temporarily keep ICU in actions_normalize.cpp and ldml_transforms.cpp - expand wasm opts in unit tests	2024-05-24 10:42:57 -05:00
Steven R. Loomis	d8b5c25954	chore(core): refactor to move most normalization calls into util_normalize.cpp ... - fold ldml_utils.hpp into core_icu.h - refactor to use km::core::util::normalize_nfd() functions where simple to do so For #9467	2024-05-23 14:47:38 -05:00