erabo/spiegel-keyman
1
0
Fork 0
mirror of https://github.com/keymanapp/keyman.git synced 2026-08-16 05:39:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
23875 commits 83 branches 3134 tags 579 MiB
Commit graph

70 commits

Author SHA1 Message Date
Steven R. Loomis
b2b897970d feat(core): stacked markers 🙀 
- renamed the single-segment functions to normalize_nfd_markers_segment() since they should only be run on a single segment
- commented out NFC for now, dead code
- add some additional checks/DebugLog around normalization

#10320
 2024-01-06 11:33:51 -06:00
Steven R. Loomis
f4be7856da feat(core): stacked markers tests 🙀 
- we can finally support single markers with the new structure
 2024-01-06 10:17:10 -06:00
Steven R. Loomis
52fe9073a0 chore(core): stacked markers tests 🙀 
- first we break

#10320
 2024-01-05 18:52:43 -06:00
Steven R. Loomis
91b761b027 feat(core): fixup markers as regex 🙀 
- move prepend_marker out of line
- replace bool for_regex with an enum
- cleanup another instance of \b -> \u0008

#9121
 2024-01-02 17:21:04 -06:00
Steven R. Loomis
02ba64d857 feat(core): fixup markers as regex 🙀 
- needed to remove/re-add the “≈≈[\u0001-\uD7FE]” format  (aka LDML_MARKER_ANY_INDEX)

#9121
 2023-12-30 21:09:42 -06:00
Steven R. Loomis
80ba5aa0ce fix(developer): escape markers as regex 🙀 
- escape markers to prevent the n'th marker from having some other meaning
 2023-12-30 11:18:51 -06:00
Steven R. Loomis
53201263d9 fix(core): use marker-safe normalization in between transform stages 🙀 
- the intermediate stages of transforms also need to use marker-safe normalization
- re-enable a test that was failing previously due to this

For: #9468
 2023-12-22 17:51:18 -06:00
Steven R. Loomis
74985f4576 fix(core): ldml: fix bad usage of unique_ptr 🙀 
- needed to declare the pointer properly

Fixes: bug(core): Build failures in LDML code on newer Ubuntu versions #10249
 2023-12-13 13:06:27 -06:00
Steven R. Loomis
7735fc045a fix(core): improve memory allocation issues in calling into icu 🙀 
- typo from  code review

For: #9121

Co-authored-by: Joshua A. Horton <joshua_horton@sil.org>
 2023-12-12 08:18:45 -06:00
Steven R. Loomis
6506bfd89d fix(core): improve memory allocation issues in calling into icu 🙀 
For: #9121
 2023-12-11 16:53:41 -06:00
Steven R. Loomis
350eb83ab3 feat(core): ldml typedef for reorder weights 🙀 
For: #9707
 2023-11-16 16:13:48 -06:00
Steven R. Loomis
3cd6c7630c feat(core): ldml tertiary reorder 🙀 
- handle case where there's no tertiary base before the first char
- separate some element API tests from the actual tertiary test.  Tests were failing because of now-correct implementation.

Fixes: #9707
 2023-11-16 13:55:49 -06:00
Steven R. Loomis
f7566c7356 feat(core): ldml tertiary reorder 🙀 
- implemented tertiary reordering
- reinstated bengali reordering!
- fixed reordering to be longest-first (by sorting the list before applying)
- updated debug logging

Fixes: #9707
 2023-11-16 12:31:08 -06:00
Steven R. Loomis
4bbafa6903 feat(core): marker normalization 🙀 
- literally a bad assert. the error case is handled below, in fact the unit test tests for it.
- for some reason, assert.h wasn't included in some cases locally.

For: #9468
 2023-11-16 12:11:41 -06:00
Steven R. Loomis
07a4821336 feat(core): marker normalization 🙀 
- a little further
- couple places where "it wasn't plugged in"
- adding some LDML-TODOs - marker creep
- fixed one unnecessary alloc/dealloc

For: #9468
 2023-11-15 17:00:04 -06:00
Steven R. Loomis
78c2ac4aec feat(core): ldml marker normalization 🙀 
- refactor out backspace processing into a function.
- for now, just drop any markers in the context when we're lopping off the end

For: #9468
 2023-11-14 07:28:41 -06:00
Steven R. Loomis
a69feb583f feat(core): ldml marker normalization 🙀 
- km::kbp is soooo last month!
- test_transforms can run NFD with markers, with some caveats.

For: #9468
 2023-11-14 07:28:10 -06:00
Steven R. Loomis
015a738226 feat(core): ldml marker normalization 🙀 
- add a new remove_markers(std::u32string) function
- add test cases for text utils
- update (failing) test cases for transform
- improve documentation of append process
- support KM_CORE_BT_UNKNOWN in ldml test
- remove_markers with a map
- update normalize test

For: #9468
 2023-11-14 07:26:47 -06:00
Eberhard Beilharz
7369b22599
chore(linux): Merge with chore/linux/9733_RenameLibknmkbp 
# Conflicts:
#	core/src/debuglog.h
#	core/src/ldml/ldml_processor.cpp
#	core/src/ldml/ldml_transforms.cpp
#	core/src/ldml/ldml_transforms.hpp
#	core/tests/unit/ldml/ldml.cpp
 2023-10-20 12:52:38 +02:00
Steven R. Loomis
4cdc2c3b54 fix(core): ldml more updates to normalization per review 🙀 
- reduce function depth
- catch another gotcha (valid was being overwritten)

For: #9468
 2023-10-19 17:10:01 -05:00
Steven R. Loomis
1d9820b020 fix(core): ldml more updates to normalization per review 🙀 
- improve assert failure handling- fix a potential heap leak

For: #9468
 2023-10-19 13:33:55 -05:00
Eberhard Beilharz
604fbc1a75
chore(linux): Rename namespace kbp to core 2023-10-19 10:39:55 +02:00
Steven R. Loomis
3723e6b3d7 fix(core): ldml more updates to normalization per review 🙀 
- consistency

For: #9468
 2023-10-18 17:58:05 -05:00
Steven R. Loomis
f1b12a1848 fix(core): ldml more updates to normalization per review 🙀 
- refactor internal normalization call

For: #9468
 2023-10-18 17:55:08 -05:00
Steven R. Loomis
9dbf70c94e fix(core): ldml more updates to normalization per review 🙀 
- verify marker string as it's popped off
- add a missing pragma to debuglog.h
- move UErrorCode out of ldml_processor, change normalization functions to return true on success

For: #9468
 2023-10-18 17:47:56 -05:00
Steven R. Loomis
9ee183c0ee fix(core): ldml update to normalization per review 🙀 
- simplify normalize_nfc and normalize_nfd functions

For: #9468
 2023-10-16 17:21:39 -05:00
Steven R. Loomis
a577bd2f53 feat(core): ldml normalization 🙀 
- more progress in the normalization pipeline. Trying to keep it from leaking.
- normalize output to NFC.
- normalize json test data to NFC (both context and expected).
- we do NOT try to normalize the 'embedded' strings currently.
- yet more test data fixes (turns out u+00e0 ≠ u+00e8)

For: #9468
 2023-10-12 16:57:37 -05:00
Steven R. Loomis
daa0523dde feat(core): ldml normalization 🙀 
- 0 steps forward, 4294967294 steps back

For: #9468
 2023-10-10 19:01:14 -05:00
Marc Durdin
ba5f18defd chore: rename km_kbp_ to km_core_ 2023-10-09 14:53:13 +07:00
Steven R. Loomis
ab29171b56 fix(core): regex set mapping support 🙀 
- fix alignment

#9121
 2023-08-24 14:09:31 -05:00
Steven R. Loomis
09772bb18d feat(core): regex set mapping support 🙀 
- passes
- added some utilities on transform_element
- added more validation of transforms
- added tests of utility functions

#9121
 2023-08-23 18:13:19 -05:00
Steven R. Loomis
935dec4304 feat(core): regex set mapping support 🙀 
- restructured some things, loading and such seems to be in good shape

#9121
 2023-08-23 15:03:44 -05:00
Steven R. Loomis
b9bf313129 feat(core): regex set mapping support 🙀 
- good failing test
 2023-08-22 20:10:42 -05:00
Steven R. Loomis
7a312520fb feat(core): regex cleanup 🙀 
- restructure to not allocate the Matcher twice, save a little processing

For: #9121
 2023-08-11 16:32:09 -05:00
Steven R. Loomis
ad03a5cdcf chore(core): regex cleanup 🙀 
- more comments

For: #9121
 2023-08-11 15:32:57 -05:00
Steven R. Loomis
75fcc5f3fe chore(core): regex cleanup 🙀 
- cache the icu::RegexPattern
- rename USet to avoid a conflict with the real one!

For: #9121
 2023-08-10 18:35:06 -05:00
Steven R. Loomis
aecb7c5f5e feat(core): regex apply-side 🙀 
- apply side is working using replaceFirst()
- mapTo is a not-yet, that will be a special case

For: #9121
 2023-08-09 18:08:25 -05:00
Steven R. Loomis
1aba900b01 feat(core): regex match-side 🙀 
- match side is working
- lots of string 'schlepping!

For: #9121
 2023-08-09 17:45:56 -05:00
Steven R. Loomis
5c38b18395 feat(core): update icu dependencies 🙀 
- enable icui18n
- .. without breaking anything else in ldml tests

For: #9121
 2023-08-08 18:45:03 -05:00
Marc Durdin
cdf026c2ba chore: Merge branch 'feature-kmc-kmw' into chore/merge-master-into-feature-kmc-kmw-a17s18 2023-08-04 12:54:10 +07:00
Marc Durdin
41aac59a4a chore(developer): minor patches to fix compile issues 
Fixes #9377.

1. ldml_transforms.cpp used emplace_back with char instead of dword
   values (#9377)
2. import was updated to src/ instead of ../ by vscode in build.ts
   (no related issue)
 2023-08-01 13:27:31 +07:00
Steven R. Loomis
99c2aff760 chore(core): turn off debugging 🙀 
- fix a stray define

For: #7375
 2023-07-26 16:18:53 -05:00
Steven R. Loomis
875d59987a chore(core): update documentation in transform logic and processor 🙀 
- drop some unneeded variables also, should be no semantic change

For: #7375
 2023-07-26 16:16:57 -05:00
Steven R. Loomis
84dc28f512 feat(core): transform/reorder 🙀 
- remove another debug print call

#7375
 2023-07-25 16:32:42 -05:00
Steven R. Loomis
f88ffbff94 chore(core): improve validity check on transform 🙀 
- also remove unnecessary loops with unused parameters

#7375
 2023-07-21 12:20:54 -05:00
Steven R. Loomis
4d9fd28181 feat(core): fix UMR in Uset 🙀 
- USet struct was depending on the COMP_KEYBOARD_EX (the .kmx data), but was used after load completed during processing
- Change the USet structure to contain a std::list<COMP_KMXPLUS_USET_RANGE> i.e. copying it
- but, while we're at it, added functions to validate that characters are within valid Unicode ranges.

#7375
 2023-07-21 11:42:32 -05:00
Steven R. Loomis
ac6dbb6291 chore(core): scale back kmxplus debugging 🙀 
- define KMXPLUS_DEBUG_LOAD, 0 by default, to get details such as 'each string' and 'each key'
- overall debugging and anomalies still DebugLog by default.
- new macro DebugLoad
- also KMXPLUS_DEBUG_TRANSFORM similarly and DebugTran
 2023-07-20 12:04:50 -05:00
Steven R. Loomis
96b11d8a73 feat(core): transform/reorder passing tests! 🙀 
- remove a debug print call

#7375
 2023-07-19 18:14:01 -05:00
Steven R. Loomis
7f07f951ae feat(core): transform/reorder passing tests! 🙀 
- reorder now handles runs, where a run starts at primary=0
- reorder still returns as if it had matched/altered the entire string

#7375
 2023-07-19 18:07:19 -05:00
Steven R. Loomis
9fc62790b9 feat(core): transform/reorder processing a little less broken 🙀 
#7375
 2023-07-19 15:29:38 -05:00