Commit graph

232 commits

Author SHA1 Message Date
Steven R. Loomis
c3bae838ec feat(developer): ldml marker normalization in ts 🙀
- use Intl.Segmenter instead.

For: #10317
2024-01-19 18:32:08 -06:00
Steven R. Loomis
72371675dd
Merge branch 'master' into feat/developer/10317-dev-nfd-epic-ldml 2024-01-19 18:01:38 -06:00
Steven R. Loomis
2bb4747c2f feat(developer): ldml marker normalization in ts 🙀
- add nfd table from ICU

For: #10317
2024-01-19 16:26:01 -06:00
Steven R. Loomis
9ff937b4ab
Apply suggestions from code review
Co-authored-by: Marc Durdin <marc@durdin.net>
2024-01-18 23:59:29 -06:00
Steven R. Loomis
7bc8831267 feat(core): updates to add_back_markers 🙀
- support out-of-order re-adding
- test for updated marker map
2024-01-17 19:04:11 -06:00
Steven R. Loomis
2e46bfea81 chore: reformat 2024-01-17 19:04:05 -06:00
Steven R. Loomis
990635eb09 feat(core): WIP attempt at cross segment markers 🙀
#10369
2024-01-17 19:03:43 -06:00
Steven R. Loomis
e4e7d9fd27
Merge branch 'feat/core/10319-escaping-you-epic-ldml' into feat/developer/10319-remove-old-escape-epic-ldml 2024-01-17 18:49:50 -06:00
Steven R. Loomis
b6a6b7e6b8 feat(core): ldml fix test case and parser 🙀
For: #10319
2024-01-17 18:32:39 -06:00
Steven R. Loomis
d08e7051cf feat(core): ldml updates per code review 🙀
- don't try to remove irregular sentinel sequences
- update test

For: #10319
2024-01-17 18:34:56 -06:00
Steven R. Loomis
eb39bba374 chore(developer,core): change sample and test files to use \u{…} 🙀
- #10389 will require this, but at least change the test files

#10319
2024-01-15 17:27:41 -06:00
Steven R. Loomis
ab67061843 feat(core): 32 bit escapades 🙀
- improved test code and marker parsing

#10319
2024-01-15 14:07:19 -06:00
Steven R. Loomis
c5d0e0525e feat(developer): 32 bit escapades 🙀
- escaping working, matching not yet

#10319
2024-01-13 12:55:51 -06:00
Steven R. Loomis
e06cf99b02
Merge branch 'feat/core/10320-stack-markers-epic-ldml' into chore/core/10320-refactor-epic-ldml 2024-01-08 12:37:51 -06:00
Steven R. Loomis
776a902eb0 feat(core): stacked markers 🙀
- build fix for win

#10320
2024-01-06 13:56:37 -06:00
Steven R. Loomis
f38054f65a chore(core): split out some files 🙀
- markers/normalization really needs its own file
- split out ldml_markers.* from ldml_transforms.*
- split out ldml_utils.hpp for a common assert macro

#10320
2024-01-06 11:56:11 -06:00
Steven R. Loomis
b2b897970d feat(core): stacked markers 🙀
- renamed the single-segment functions to normalize_nfd_markers_segment() since they should only be run on a single segment
- commented out NFC for now, dead code
- add some additional checks/DebugLog around normalization

#10320
2024-01-06 11:33:51 -06:00
Steven R. Loomis
f4be7856da feat(core): stacked markers tests 🙀
- we can finally support single markers with the new structure
2024-01-06 10:17:10 -06:00
Steven R. Loomis
52fe9073a0 chore(core): stacked markers tests 🙀
- first we break

#10320
2024-01-05 18:52:43 -06:00
Steven R. Loomis
91b761b027 feat(core): fixup markers as regex 🙀
- move prepend_marker out of line
- replace bool for_regex with an enum
- cleanup another instance of \b -> \u0008

#9121
2024-01-02 17:21:04 -06:00
Steven R. Loomis
a6151e681b
Merge branch 'chore/core/ldml-test-robust-epic-ldml' into feat/developer/9121-reject-bad-regex-epic-ldml 2024-01-02 17:04:37 -06:00
Steven R. Loomis
84c218ff85
Merge branch 'feat/core/9121-more-regex-epic-ldml' into chore/core/ldml-test-robust-epic-ldml 2024-01-02 17:03:07 -06:00
Steven R. Loomis
0c3d1b513e
Merge branch 'feat/core/9468-stacked-markers-epic-ldml' into feat/core/9121-more-regex-epic-ldml 2024-01-02 17:03:00 -06:00
Steven R. Loomis
32ba632af9
Update core/tests/unit/ldml/test_transforms.cpp
Co-authored-by: Marc Durdin <marc@durdin.net>
2024-01-02 17:02:35 -06:00
Steven R. Loomis
02ba64d857 feat(core): fixup markers as regex 🙀
- needed to remove/re-add the “≈≈[\u0001-\uD7FE]” format  (aka LDML_MARKER_ANY_INDEX)

#9121
2023-12-30 21:09:42 -06:00
Steven R. Loomis
80ba5aa0ce fix(developer): escape markers as regex 🙀
- escape markers to prevent the n'th marker from having some other meaning
2023-12-30 11:18:51 -06:00
Steven R. Loomis
763f7abeb0 chore(core): failing test with markers 🙀
- marker overrun! the 40th marker becomes a parenthesis, breaking regex

feat(core): ldml proper regex implementation 🙀🙀 #9121
2023-12-28 18:57:33 -06:00
Steven R. Loomis
561cecb0de fix(core): fix ldml test runner loops 🙀
- more easily handle test events
2023-12-28 17:57:40 -06:00
Steven R. Loomis
ae96008c11 feat(core): ldml more regex tests 🙀
- match in a set
- quantifiers

For: #9121
2023-12-28 17:38:21 -06:00
Steven R. Loomis
ef2f8395b4 feat(core): better reporting in ldml tests 🙀
- more err reporting out of inner functions such as key not found
- helper functions for setting FAIL and SKIP actions with messages
- colorization

For: #9121
2023-12-28 17:38:21 -06:00
Steven R. Loomis
53201263d9 fix(core): use marker-safe normalization in between transform stages 🙀
- the intermediate stages of transforms also need to use marker-safe normalization
- re-enable a test that was failing previously due to this

For: #9468
2023-12-22 17:51:18 -06:00
Steven R. Loomis
9d1599d557 chore(core): add some more marker tests 🙀
- one disabled for now

For #9468
2023-12-21 11:36:35 -06:00
Steven R. Loomis
12aa9a5b69 chore(core): add some more marker tests 🙀
For #9468
2023-12-20 19:23:29 -06:00
Steven R. Loomis
0b3e7a94e8 chore(core): uncomment some more tests 🙀
For #9468
2023-12-20 18:50:21 -06:00
Steven R. Loomis
0d3458deef chore(core): comments in key-not-found 🙀
- typo fix
- clarify default behavior for missing keys

For: #9451
2023-12-12 18:43:00 -06:00
Steven R. Loomis
1e32a2a963 feat(core): ldml improve key-not-found 🙀
- copy exception table for reset from VKScanCodes.cpp
- update test

For: #9451
2023-12-07 17:15:44 -06:00
Steven R. Loomis
392c535270 feat(core): ldml improve key-not-found 🙀
- also affects markers, feat(core): normalization per spec for transforms/etc 🙀  #9468
- keep markers in nfd context string
- fix ldml test harness to handle context reset
- update test case
- still issues with overproduction of markers in the context

For: #9451
2023-12-06 18:33:51 -06:00
Steven R. Loomis
11334ed7fa feat(core): ldml improve key-not-found 🙀
- distinguish between unmapped and 0-length output strings
- don't do any processing for 0-length output strings
- no change to developer, that will be next
- remove a comment referencing transform=no

For: #9451
2023-11-28 18:07:46 -06:00
Steven R. Loomis
a9b2dc9ede
Merge branch 'fix/developer/9438-vis-kbd-crash-epic-ldml' into fix/core-9961-optional-startContext-epic-ldml 2023-11-21 18:23:04 -06:00
Steven R. Loomis
037454b86e
Merge branch 'master' into feat/core/9707-tertiary-reorder-epic-ldml 2023-11-21 16:43:57 -06:00
Steven R. Loomis
d82f64fadc
Merge branch 'master' into feat/core/9450-backspace-epic-ldml 2023-11-21 15:01:51 -06:00
Steven R. Loomis
f524519b3f feat(developer): ldml fix testcase processing 🙀
- KM_CORE_BT_UNKNOWN should only be used with an empty context

For: #9468  but related to #9450
2023-11-20 16:57:47 -06:00
Steven R. Loomis
86e8a4f609
Merge branch 'fix/developer/9438-vis-kbd-crash-epic-ldml' into fix/core-9961-optional-startContext-epic-ldml 2023-11-16 16:17:12 -06:00
Steven R. Loomis
350eb83ab3 feat(core): ldml typedef for reorder weights 🙀
For: #9707
2023-11-16 16:13:48 -06:00
Steven R. Loomis
6bc72ff294 fix(core): dx: ldml: startContext should be optional in ldml test data 🙀
- per spec
- update tests to remove these

Fixes: #9961
2023-11-16 15:51:56 -06:00
Steven R. Loomis
3cd6c7630c feat(core): ldml tertiary reorder 🙀
- handle case where there's no tertiary base before the first char
- separate some element API tests from the actual tertiary test.  Tests were failing because of now-correct implementation.

Fixes: #9707
2023-11-16 13:55:49 -06:00
Steven R. Loomis
30d951ef67 feat(core): ldml dx: launch ldml unit tests before keyboard tests 🙀
For: #9707
2023-11-16 13:43:12 -06:00
Steven R. Loomis
f7566c7356 feat(core): ldml tertiary reorder 🙀
- implemented tertiary reordering
- reinstated bengali reordering!
- fixed reordering to be longest-first (by sorting the list before applying)
- updated debug logging

Fixes: #9707
2023-11-16 12:31:08 -06:00
Steven R. Loomis
85a5afafc9 feat(core): ldml tertiary reorder 🙀
- failing test cases, hooray! In XML and C++

For: #9707
2023-11-16 12:31:08 -06:00
Steven R. Loomis
d0c8ea5c88 feat(core): ldml backspace transform 🙀
- refactor keypress some (needs more)
- copypasta implementation of backspace transform

For: #9450
2023-11-16 12:25:15 -06:00