Commit graph

337 commits

Author SHA1 Message Date
Steven R. Loomis
3cd6c7630c feat(core): ldml tertiary reorder 🙀
- handle case where there's no tertiary base before the first char
- separate some element API tests from the actual tertiary test.  Tests were failing because of now-correct implementation.

Fixes: #9707
2023-11-16 13:55:49 -06:00
Steven R. Loomis
30d951ef67 feat(core): ldml dx: launch ldml unit tests before keyboard tests 🙀
For: #9707
2023-11-16 13:43:12 -06:00
Steven R. Loomis
f7566c7356 feat(core): ldml tertiary reorder 🙀
- implemented tertiary reordering
- reinstated bengali reordering!
- fixed reordering to be longest-first (by sorting the list before applying)
- updated debug logging

Fixes: #9707
2023-11-16 12:31:08 -06:00
Steven R. Loomis
85a5afafc9 feat(core): ldml tertiary reorder 🙀
- failing test cases, hooray! In XML and C++

For: #9707
2023-11-16 12:31:08 -06:00
Steven R. Loomis
d0c8ea5c88 feat(core): ldml backspace transform 🙀
- refactor keypress some (needs more)
- copypasta implementation of backspace transform

For: #9450
2023-11-16 12:25:15 -06:00
Steven R. Loomis
e28f35903a feat(core): marker normalization 🙀
- test fix

For: #9468
2023-11-15 17:29:18 -06:00
Steven R. Loomis
5a04a812fa feat(core): marker normalization 🙀
- temporarily remove a backspace test until #9450 is fixed.

For: #9468
2023-11-15 17:09:26 -06:00
Steven R. Loomis
07a4821336 feat(core): marker normalization 🙀
- a little further
- couple places where "it wasn't plugged in"
- adding some LDML-TODOs - marker creep
- fixed one unnecessary alloc/dealloc

For: #9468
2023-11-15 17:00:04 -06:00
Steven R. Loomis
2f843d98f3 feat(core): marker normalization 🙀
- go back to NFD for the context, for now
- anticipating when the privatecontext is NFD but the public context is NFC
- also update the test cases

For: #9468
2023-11-14 16:36:15 -06:00
Steven R. Loomis
7b481945e6 feat(core): test fix 🙀
- fix a cast
- fix some test cases

For: #9468
2023-11-14 07:28:57 -06:00
Steven R. Loomis
78c2ac4aec feat(core): ldml marker normalization 🙀
- refactor out backspace processing into a function.
- for now, just drop any markers in the context when we're lopping off the end

For: #9468
2023-11-14 07:28:41 -06:00
Steven R. Loomis
72830edfad Apply suggestions from code review
Co-authored-by: Marc Durdin <marc@durdin.net>
2023-11-14 07:28:41 -06:00
Steven R. Loomis
a69feb583f feat(core): ldml marker normalization 🙀
- km::kbp is soooo last month!
- test_transforms can run NFD with markers, with some caveats.

For: #9468
2023-11-14 07:28:10 -06:00
Steven R. Loomis
015a738226 feat(core): ldml marker normalization 🙀
- add a new remove_markers(std::u32string) function
- add test cases for text utils
- update (failing) test cases for transform
- improve documentation of append process
- support KM_CORE_BT_UNKNOWN in ldml test
- remove_markers with a map
- update normalize test

For: #9468
2023-11-14 07:26:47 -06:00
Steven R. Loomis
255960ecae feat(core): ldml dx: dump vkey and modifier 🙀
For: #9468
2023-11-14 07:26:47 -06:00
Steven R. Loomis
efc8ef022f feat(core): ldml normalization 🙀
- some failing marker tests

For: #9468
2023-11-14 07:26:47 -06:00
Steven R. Loomis
09411eba33 feat(common): ldml update to WIP cldr data 🙀
- bn-bengali with a basic test
- fr-optimise
- note: disabling bengali reorders for now, not working properly due to tertiary reordering feat(core): tertiary reorders 🙀  #9707

For: #9916
2023-11-02 17:44:14 -05:00
Steven R. Loomis
a45dc5c345 chore(developer): ldml update to prefinal v44 🙀
-  fix binary size issue
- upstream: issue in abnt2 sample keyboard

For feat(developer): ldml re-sync with CLDR v44 spec 🙀  #9838
2023-10-26 18:21:56 -05:00
Steven R. Loomis
4d358cabda chore(developer): ldml update to prefinal v44 🙀
-  utilities for extracting key ids
-  more key validation and munging
-  touch layout
-  fix basic.txt

For feat(developer): ldml re-sync with CLDR v44 spec 🙀  #9838
2023-10-26 12:57:07 -05:00
Steven R. Loomis
870cf76049 chore(developer,core): ldml update to prefinal v44 🙀
- more data files

For feat(developer): ldml re-sync with CLDR v44 spec 🙀  #9838
2023-10-24 14:56:09 -05:00
Steven R. Loomis
5dbda001fc chore(common,core): ldml update to prefinal v44 🙀
- common passes tests
- updated data files

For feat(developer): ldml re-sync with CLDR v44 spec 🙀  #9838
2023-10-24 14:47:22 -05:00
Eberhard Beilharz
f4d8529789
chore(linux): rename keyboardprocessor_ldml.* to keyman_core_ldml 2023-10-20 17:30:10 +02:00
Eberhard Beilharz
38e5d34459
chore(linux): Rename namespace
Missed a few occurrences that sneaked in after merging with latest master.
2023-10-20 17:28:37 +02:00
Eberhard Beilharz
7369b22599
chore(linux): Merge with chore/linux/9733_RenameLibknmkbp
# Conflicts:
#	core/src/debuglog.h
#	core/src/ldml/ldml_processor.cpp
#	core/src/ldml/ldml_transforms.cpp
#	core/src/ldml/ldml_transforms.hpp
#	core/tests/unit/ldml/ldml.cpp
2023-10-20 12:52:38 +02:00
Steven R. Loomis
13eed28a36 Merge branch 'feat/core/9468-normalization-epic-ldml' into feat/core/9468-test-improvement-epic-ldml 2023-10-19 14:00:38 -05:00
Eberhard Beilharz
604fbc1a75
chore(linux): Rename namespace kbp to core 2023-10-19 10:39:55 +02:00
Steven R. Loomis
9dbf70c94e fix(core): ldml more updates to normalization per review 🙀
- verify marker string as it's popped off
- add a missing pragma to debuglog.h
- move UErrorCode out of ldml_processor, change normalization functions to return true on success

For: #9468
2023-10-18 17:47:56 -05:00
Steven R. Loomis
55bf0642a5 feat(core): add backspace test 🙀
- test needs to be "t" not "ta"

For: #9760
2023-10-13 16:56:00 -05:00
Steven R. Loomis
6a62d01481 chore(core): dx: ldml test subselection 🙀
- add a filter option to run just one subtest
- support backspace event, revamp test JSON format

For: #9468
2023-10-13 16:45:56 -05:00
Steven R. Loomis
bf15feb116 feat(core): ldml normalization 🙀
- fix int -> auto

For: #9468
2023-10-12 19:13:52 -05:00
Steven R. Loomis
95ad1c47e4 feat(core): ldml normalization 🙀
- fix unnecessary test churn.

For: #9468
2023-10-12 17:46:41 -05:00
Steven R. Loomis
84b4836bbf
Merge branch 'master' into feat/core/9468-normalization-epic-ldml 2023-10-12 16:59:18 -05:00
Steven R. Loomis
a577bd2f53 feat(core): ldml normalization 🙀
- more progress in the normalization pipeline. Trying to keep it from leaking.
- normalize output to NFC.
- normalize json test data to NFC (both context and expected).
- we do NOT try to normalize the 'embedded' strings currently.
- yet more test data fixes (turns out u+00e0 ≠ u+00e8)

For: #9468
2023-10-12 16:57:37 -05:00
Steven R. Loomis
736af4b957 feat(core): ldml normalization 🙀
- change more tests to NFD for now

For: #9468
2023-10-12 15:12:25 -05:00
Steven R. Loomis
522d2ef583 feat(core): ldml normalization 🙀
- change tests to NFD for now

For: #9468
2023-10-12 10:10:37 -05:00
Marc Durdin
f61e65ceb9
Merge pull request #9736 from keymanapp/chore/merge-master-into-package-metadata
chore: merge master into package-metadata
2023-10-11 11:28:07 +11:00
Steven R. Loomis
daa0523dde feat(core): ldml normalization 🙀
- 0 steps forward, 4294967294 steps back

For: #9468
2023-10-10 19:01:14 -05:00
Steven R. Loomis
15e6fb60db chore(core): dx: improve test output 🙀
- colorize ldml test

For: #9468
2023-10-10 17:51:12 -05:00
Steven R. Loomis
8c4642ace3
Merge branch 'master' into feat/core/9468-normalization-epic-ldml 2023-10-10 17:16:42 -05:00
Steven R. Loomis
dcb6796d5b chore(core): Some failing normalization transform tests 🙀
For: #9468
2023-10-09 18:45:17 -05:00
Marc Durdin
db76e2541d chore: rename KM_KBP_ to KM_CORE_ 2023-10-09 14:53:22 +07:00
Marc Durdin
ba5f18defd chore: rename km_kbp_ to km_core_ 2023-10-09 14:53:13 +07:00
Marc Durdin
4ef18f5e29 chore(core): rename keyboardprocessor.h to keyman_core_api.h
Fixes #9721.

Also tweaks some documentation references to keyboardprocessor, but does
not touch those relating to debian/control.
2023-10-09 14:21:01 +07:00
Steven R. Loomis
8f4be0c592 feat(core): more marker 🙀
- reenable some tests

For: #9119
2023-10-04 18:45:45 -05:00
Steven R. Loomis
2472a941cf feat(core): match any marker 🙀
- add a regex form that matches any marker

For: #9119
2023-10-04 18:34:20 -05:00
Marc Durdin
693aee5c99
Merge branch 'epic/package-metadata' into chore/merge-master-into-package-metadata 2023-10-04 19:34:16 +11:00
Marc Durdin
e0c2778199
Merge pull request #9644 from keymanapp/chore/merge-master-into-package-metadata
chore: merge master into package-metadata 🎺
2023-10-04 03:43:45 +02:00
Marc Durdin
f864f57d8b
Merge pull request #9607 from keymanapp/feat/developer/kmc-package-windows-installer
feat(developer): Windows package installer compiler 🎺
2023-10-03 15:35:43 +02:00
Steven R. Loomis
64eaf39bcb chore(core): use sourcemaps when calling kmc for tests 2023-09-30 18:13:47 -05:00
Steven R. Loomis
5fa99f5314 fix(common): update keyman for removal of vkeys 🙀
dropped in upstream CLDR-17093

- missed changes

See #7135
2023-09-28 13:32:02 -05:00