erabo/spiegel-keyman
1
0
Fork 0
mirror of https://github.com/keymanapp/keyman.git synced 2026-08-16 05:39:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
26561 commits 83 branches 3134 tags 579 MiB
Commit graph

83 commits

Author SHA1 Message Date
Steven R. Loomis
d8b5c25954 chore(core): refactor to move most normalization calls into util_normalize.cpp 
- fold ldml_utils.hpp into core_icu.h
- refactor to use km::core::util::normalize_nfd() functions where simple to do so

For #9467
 2024-05-23 14:47:38 -05:00
Steven R. Loomis
2ee402c2b8
Update core/src/ldml/ldml_transforms.cpp 
Co-authored-by: Marc Durdin <marc@durdin.net>
 2024-02-19 12:37:26 -06:00
Steven R. Loomis
6b229ffa14 fix(core): fix output append logic in any_group::apply_transform 
- logic updating intermediate 'output' was incorrect
- update comments
- also replace size() with length() several places

Fixes #10736
 2024-02-17 23:50:55 -06:00
Steven R. Loomis
ceb990e713 chore(core): more shuffling 
For #10736
 2024-02-16 18:27:35 -06:00
Steven R. Loomis
746af68afa chore(core): shuffle transform matcher. re-enable failing case 
For #10736
 2024-02-16 12:49:19 -06:00
Steven R. Loomis
e7b31113a3 chore(core): remove obsolete TODOs. Temporary change to test. 
- TODO: 10734 There's a bug in the matching still. Come back and fix this, but the crasher is fixed.

For #10734
 2024-02-15 16:21:53 -06:00
Steven R. Loomis
5edb70f07a fix(core): fix for ldml_transforms.cpp 
For #10734
 2024-02-15 15:26:30 -06:00
Steven R. Loomis
71443836a2 feat(developer): support normalization=disabled 🙀 
- avoid dead store to transforms class

#10554
 2024-02-01 19:05:31 -06:00
Steven R. Loomis
7572c9476e feat(developer): support normalization=disabled 🙀 
- main core changes and test changes

#10554
 2024-02-01 17:32:42 -06:00
Steven R. Loomis
2c92cfea4b feat(core): reorder processing 🙀 
- update per code review
- move quickcheck of map into add_back_markers()

For: #10516
 2024-01-31 17:44:21 -06:00
Steven R. Loomis
b429dc3383 feat(core): markers and reorder 🙀 
- reorders don't interact with markers, but must transit them
- add updated tests
- simplify reorder_group::apply(): the 'common prefix' discussion
was out-of-date as std::mismatch is handled by the ldml_processor
and the core context.
- add_back_markers() promoted to SPI so it can be called from reordering.

Fixes: #10516
 2024-01-31 17:44:21 -06:00
Steven R. Loomis
3ccffe9430 feat(core): ldml marker: refactor, use nfd props 🙀 
- split out parse_next_marker()
- use NFD safe boundaries to segment marker interaction

For: #10369
 2024-01-18 13:51:40 -06:00
Steven R. Loomis
f38054f65a chore(core): split out some files 🙀 
- markers/normalization really needs its own file
- split out ldml_markers.* from ldml_transforms.*
- split out ldml_utils.hpp for a common assert macro

#10320
 2024-01-06 11:56:11 -06:00
Steven R. Loomis
b2b897970d feat(core): stacked markers 🙀 
- renamed the single-segment functions to normalize_nfd_markers_segment() since they should only be run on a single segment
- commented out NFC for now, dead code
- add some additional checks/DebugLog around normalization

#10320
 2024-01-06 11:33:51 -06:00
Steven R. Loomis
f4be7856da feat(core): stacked markers tests 🙀 
- we can finally support single markers with the new structure
 2024-01-06 10:17:10 -06:00
Steven R. Loomis
52fe9073a0 chore(core): stacked markers tests 🙀 
- first we break

#10320
 2024-01-05 18:52:43 -06:00
Steven R. Loomis
91b761b027 feat(core): fixup markers as regex 🙀 
- move prepend_marker out of line
- replace bool for_regex with an enum
- cleanup another instance of \b -> \u0008

#9121
 2024-01-02 17:21:04 -06:00
Steven R. Loomis
02ba64d857 feat(core): fixup markers as regex 🙀 
- needed to remove/re-add the “≈≈[\u0001-\uD7FE]” format  (aka LDML_MARKER_ANY_INDEX)

#9121
 2023-12-30 21:09:42 -06:00
Steven R. Loomis
80ba5aa0ce fix(developer): escape markers as regex 🙀 
- escape markers to prevent the n'th marker from having some other meaning
 2023-12-30 11:18:51 -06:00
Steven R. Loomis
53201263d9 fix(core): use marker-safe normalization in between transform stages 🙀 
- the intermediate stages of transforms also need to use marker-safe normalization
- re-enable a test that was failing previously due to this

For: #9468
 2023-12-22 17:51:18 -06:00
Steven R. Loomis
74985f4576 fix(core): ldml: fix bad usage of unique_ptr 🙀 
- needed to declare the pointer properly

Fixes: bug(core): Build failures in LDML code on newer Ubuntu versions #10249
 2023-12-13 13:06:27 -06:00
Steven R. Loomis
7735fc045a fix(core): improve memory allocation issues in calling into icu 🙀 
- typo from  code review

For: #9121

Co-authored-by: Joshua A. Horton <joshua_horton@sil.org>
 2023-12-12 08:18:45 -06:00
Steven R. Loomis
6506bfd89d fix(core): improve memory allocation issues in calling into icu 🙀 
For: #9121
 2023-12-11 16:53:41 -06:00
Steven R. Loomis
350eb83ab3 feat(core): ldml typedef for reorder weights 🙀 
For: #9707
 2023-11-16 16:13:48 -06:00
Steven R. Loomis
3cd6c7630c feat(core): ldml tertiary reorder 🙀 
- handle case where there's no tertiary base before the first char
- separate some element API tests from the actual tertiary test.  Tests were failing because of now-correct implementation.

Fixes: #9707
 2023-11-16 13:55:49 -06:00
Steven R. Loomis
f7566c7356 feat(core): ldml tertiary reorder 🙀 
- implemented tertiary reordering
- reinstated bengali reordering!
- fixed reordering to be longest-first (by sorting the list before applying)
- updated debug logging

Fixes: #9707
 2023-11-16 12:31:08 -06:00
Steven R. Loomis
4bbafa6903 feat(core): marker normalization 🙀 
- literally a bad assert. the error case is handled below, in fact the unit test tests for it.
- for some reason, assert.h wasn't included in some cases locally.

For: #9468
 2023-11-16 12:11:41 -06:00
Steven R. Loomis
07a4821336 feat(core): marker normalization 🙀 
- a little further
- couple places where "it wasn't plugged in"
- adding some LDML-TODOs - marker creep
- fixed one unnecessary alloc/dealloc

For: #9468
 2023-11-15 17:00:04 -06:00
Steven R. Loomis
78c2ac4aec feat(core): ldml marker normalization 🙀 
- refactor out backspace processing into a function.
- for now, just drop any markers in the context when we're lopping off the end

For: #9468
 2023-11-14 07:28:41 -06:00
Steven R. Loomis
a69feb583f feat(core): ldml marker normalization 🙀 
- km::kbp is soooo last month!
- test_transforms can run NFD with markers, with some caveats.

For: #9468
 2023-11-14 07:28:10 -06:00
Steven R. Loomis
015a738226 feat(core): ldml marker normalization 🙀 
- add a new remove_markers(std::u32string) function
- add test cases for text utils
- update (failing) test cases for transform
- improve documentation of append process
- support KM_CORE_BT_UNKNOWN in ldml test
- remove_markers with a map
- update normalize test

For: #9468
 2023-11-14 07:26:47 -06:00
Eberhard Beilharz
7369b22599
chore(linux): Merge with chore/linux/9733_RenameLibknmkbp 
# Conflicts:
#	core/src/debuglog.h
#	core/src/ldml/ldml_processor.cpp
#	core/src/ldml/ldml_transforms.cpp
#	core/src/ldml/ldml_transforms.hpp
#	core/tests/unit/ldml/ldml.cpp
 2023-10-20 12:52:38 +02:00
Steven R. Loomis
4cdc2c3b54 fix(core): ldml more updates to normalization per review 🙀 
- reduce function depth
- catch another gotcha (valid was being overwritten)

For: #9468
 2023-10-19 17:10:01 -05:00
Steven R. Loomis
1d9820b020 fix(core): ldml more updates to normalization per review 🙀 
- improve assert failure handling- fix a potential heap leak

For: #9468
 2023-10-19 13:33:55 -05:00
Eberhard Beilharz
604fbc1a75
chore(linux): Rename namespace kbp to core 2023-10-19 10:39:55 +02:00
Steven R. Loomis
3723e6b3d7 fix(core): ldml more updates to normalization per review 🙀 
- consistency

For: #9468
 2023-10-18 17:58:05 -05:00
Steven R. Loomis
f1b12a1848 fix(core): ldml more updates to normalization per review 🙀 
- refactor internal normalization call

For: #9468
 2023-10-18 17:55:08 -05:00
Steven R. Loomis
9dbf70c94e fix(core): ldml more updates to normalization per review 🙀 
- verify marker string as it's popped off
- add a missing pragma to debuglog.h
- move UErrorCode out of ldml_processor, change normalization functions to return true on success

For: #9468
 2023-10-18 17:47:56 -05:00
Steven R. Loomis
9ee183c0ee fix(core): ldml update to normalization per review 🙀 
- simplify normalize_nfc and normalize_nfd functions

For: #9468
 2023-10-16 17:21:39 -05:00
Steven R. Loomis
a577bd2f53 feat(core): ldml normalization 🙀 
- more progress in the normalization pipeline. Trying to keep it from leaking.
- normalize output to NFC.
- normalize json test data to NFC (both context and expected).
- we do NOT try to normalize the 'embedded' strings currently.
- yet more test data fixes (turns out u+00e0 ≠ u+00e8)

For: #9468
 2023-10-12 16:57:37 -05:00
Steven R. Loomis
daa0523dde feat(core): ldml normalization 🙀 
- 0 steps forward, 4294967294 steps back

For: #9468
 2023-10-10 19:01:14 -05:00
Marc Durdin
ba5f18defd chore: rename km_kbp_ to km_core_ 2023-10-09 14:53:13 +07:00
Steven R. Loomis
ab29171b56 fix(core): regex set mapping support 🙀 
- fix alignment

#9121
 2023-08-24 14:09:31 -05:00
Steven R. Loomis
09772bb18d feat(core): regex set mapping support 🙀 
- passes
- added some utilities on transform_element
- added more validation of transforms
- added tests of utility functions

#9121
 2023-08-23 18:13:19 -05:00
Steven R. Loomis
935dec4304 feat(core): regex set mapping support 🙀 
- restructured some things, loading and such seems to be in good shape

#9121
 2023-08-23 15:03:44 -05:00
Steven R. Loomis
b9bf313129 feat(core): regex set mapping support 🙀 
- good failing test
 2023-08-22 20:10:42 -05:00
Steven R. Loomis
7a312520fb feat(core): regex cleanup 🙀 
- restructure to not allocate the Matcher twice, save a little processing

For: #9121
 2023-08-11 16:32:09 -05:00
Steven R. Loomis
ad03a5cdcf chore(core): regex cleanup 🙀 
- more comments

For: #9121
 2023-08-11 15:32:57 -05:00
Steven R. Loomis
75fcc5f3fe chore(core): regex cleanup 🙀 
- cache the icu::RegexPattern
- rename USet to avoid a conflict with the real one!

For: #9121
 2023-08-10 18:35:06 -05:00
Steven R. Loomis
aecb7c5f5e feat(core): regex apply-side 🙀 
- apply side is working using replaceFirst()
- mapTo is a not-yet, that will be a special case

For: #9121
 2023-08-09 18:08:25 -05:00