From 5510321e04ee3fb261a167d4a7967140157ca04e Mon Sep 17 00:00:00 2001 From: "Steven R. Loomis" Date: Wed, 5 Jul 2023 18:16:09 -0500 Subject: [PATCH 1/3] =?UTF-8?q?spec(core):=20marker=20spec=20=F0=9F=99=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - first cut Fixes: #9118 --- core/src/ldml/C9134_ldml_markers.md | 73 +++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 core/src/ldml/C9134_ldml_markers.md diff --git a/core/src/ldml/C9134_ldml_markers.md b/core/src/ldml/C9134_ldml_markers.md new file mode 100644 index 0000000000..3f444d5403 --- /dev/null +++ b/core/src/ldml/C9134_ldml_markers.md @@ -0,0 +1,73 @@ +# LDML Markers in Keyman + +## Background + +- This is the proposed implementation of markers for Keyman + +> Markers are placeholders which record some state, but without producing normal visible text output. They were designed particularly to support dead-keys. + +- See [tr35-keyboards-markers][] for source spec. + - [CLDR-16837][] has been filed to capture feedback on the marker spec, please add to that with any commnets. + +### Format + +- The general format is `\m{id}` where `id` is presumed to be `[0-9A-Za-z_]{1,32}` +- `\m{.}` in matching matches a single marker. + +### Location + +Markers can appear in both 'emitting' and 'matching-only' areas: + +#### Emitting + +- `key to=` for emitting markers +- `transform to=` to emit markers +- `string value=` for matching or emitting markers + +#### Match only + +- `transform from=` to match markers +- `transform after=` to match markers +- `display to=` for matching keys which contain markers + +## Theory / Encoding + +- Keyman already uses U_SENTINEL `U+FFFF` (noncharacter) +- The general proposal here is to use the sequence `U+FFFF U+EXXX` to represent marker #XXX +- `U+FFFF` cannot otherwise occur in text, so it is unique +- `U+EXXX` is always a private use character, so the marker number cannot collide with non-PUA text, however it may certainly collide with PUA text. The intention here is to reduce this possibility of collision, at least when (a) humans read a binary stream or (b) human _error_ causes the marker to show up in acutal text. As a counter example, if we used `U+FFFF U+0022` to indicate marker 0x22, then the marker might show up as a doublequote (`"`). +- `U+FFFF U+EFFF` to indicate 'any marker' corresponds to `\m{.}` +- this scheme allows for 4,095 (0xFFF) unique markers, from `U+FFFF U+E000` through `U+FFFF U+EFFE` + +## Compiler (kmc) + +- `U+FFFF` needs to be illegal as a literal or escaped sequence. So `\u{FFFF}` is not allowed, for example, nor as a literal in the UTF-8 .xml stream. + +### `vars` + +- `vars` compiler will prescan all of the elements listed under [Emitting](#emitting), but not match-only. +- All found markers will be added to a single master `list`, which will be added under the `vars.markers` section. +- Subsequent processes/compilers will be able to check this list for any missing markers. For example, a `` can then trigger an error failing validation, because nothing is defined whith emits that marker. + +### Other sections + +- `string value='\m{…}'` will simply store `\m{…}`, for application when expanded as with other variables. +- Other emitters, such as `key`, `transform` will include the string `U+FFFF U+EXXX` where XXX corresponds to the marker's number. +- Transforms will need to match against the marker or markers desired, so may need to emit sequences such as `(?:\uFFFF\uE123)` meaning a match to marker #0x123 +- matching `\m{.}` may need to expand to `(?:\uFFFF[\uE000-\uEFFE])` + +## Binary (.kmx plus) + +- The `vars.markers` is a pointer into the `list` section with a list (binary order) of the marker names +- Other strings will be in `U+FFFF U+E123` form etc. as if it was in the original text stream as such. + +## Implementation (core) + +- Core needs to recognize `U+FFFF …` sequences and convert them to markers in the context stream. +- For normal processing, Core does _not_ need to correlate the marker _number_ with a marker _id_, although this would be helpful for a debugging or tracing facility. I.e. `U+FFFF U+E123` corresponding to entry 0x123 in the `vars.markers` -> `list` table. +- Core needs to remove `U+FFFF …` sequences before they are passed to the OS. +- The default backspace processing needs to ignore `U+FFFF …` markers as it is deleting. + + +[tr35-keyboards-markers]: https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-keyboards.md#markers +[CLDR-16837]: https://unicode-org.atlassian.net/browse/CLDR-16837 From 4c096b75745544d29b003183365c5938db89064c Mon Sep 17 00:00:00 2001 From: "Steven R. Loomis" Date: Thu, 6 Jul 2023 16:32:56 -0500 Subject: [PATCH 2/3] Update core/src/ldml/C9134_ldml_markers.md Co-authored-by: Joshua Horton --- core/src/ldml/C9134_ldml_markers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/src/ldml/C9134_ldml_markers.md b/core/src/ldml/C9134_ldml_markers.md index 3f444d5403..876f30b49d 100644 --- a/core/src/ldml/C9134_ldml_markers.md +++ b/core/src/ldml/C9134_ldml_markers.md @@ -47,7 +47,7 @@ Markers can appear in both 'emitting' and 'matching-only' areas: - `vars` compiler will prescan all of the elements listed under [Emitting](#emitting), but not match-only. - All found markers will be added to a single master `list`, which will be added under the `vars.markers` section. -- Subsequent processes/compilers will be able to check this list for any missing markers. For example, a `` can then trigger an error failing validation, because nothing is defined whith emits that marker. +- Subsequent processes/compilers will be able to check this list for any missing markers. For example, a `` can then trigger an error failing validation, because nothing is defined which emits that marker. ### Other sections From 4950f741f8a3d32014942c25766dec3c9bdccf58 Mon Sep 17 00:00:00 2001 From: "Steven R. Loomis" Date: Thu, 6 Jul 2023 16:50:35 -0500 Subject: [PATCH 3/3] =?UTF-8?q?spec(core):=20marker=20spec=20=F0=9F=99=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - switch from PUA to U+0001 based indices - add some notes from comments (discussion ongoing) --- core/src/ldml/C9134_ldml_markers.md | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/core/src/ldml/C9134_ldml_markers.md b/core/src/ldml/C9134_ldml_markers.md index 876f30b49d..7a0f5d6959 100644 --- a/core/src/ldml/C9134_ldml_markers.md +++ b/core/src/ldml/C9134_ldml_markers.md @@ -33,11 +33,15 @@ Markers can appear in both 'emitting' and 'matching-only' areas: ## Theory / Encoding - Keyman already uses U_SENTINEL `U+FFFF` (noncharacter) -- The general proposal here is to use the sequence `U+FFFF U+EXXX` to represent marker #XXX +- The general proposal here is to use the sequence `U+FFFF U+XXXX` to represent marker #XXXX (starting with `U+0001`) - `U+FFFF` cannot otherwise occur in text, so it is unique -- `U+EXXX` is always a private use character, so the marker number cannot collide with non-PUA text, however it may certainly collide with PUA text. The intention here is to reduce this possibility of collision, at least when (a) humans read a binary stream or (b) human _error_ causes the marker to show up in acutal text. As a counter example, if we used `U+FFFF U+0022` to indicate marker 0x22, then the marker might show up as a doublequote (`"`). -- `U+FFFF U+EFFF` to indicate 'any marker' corresponds to `\m{.}` -- this scheme allows for 4,095 (0xFFF) unique markers, from `U+FFFF U+E000` through `U+FFFF U+EFFE` +- `U+FFFF U+FFFF` to indicate 'any marker' corresponds to `\m{.}` +- This scheme allows for 65,534 (0xFFFE) unique markers, from `U+FFFF U+0001` through `U+FFFF U+FFFE` + +## Terminology +- A marker's "number" is its position in the `markers` list, starting at index 1 (U+0001) being the first element in that list. +Note that this is different from other 0-based indices in KMX+. If there are three markers in a keyboard file, they will be numbered 1, 2, 3. +- A marker's "id" is its string name such as `a` or `acute` ## Compiler (kmc) @@ -52,21 +56,24 @@ Markers can appear in both 'emitting' and 'matching-only' areas: ### Other sections - `string value='\m{…}'` will simply store `\m{…}`, for application when expanded as with other variables. -- Other emitters, such as `key`, `transform` will include the string `U+FFFF U+EXXX` where XXX corresponds to the marker's number. -- Transforms will need to match against the marker or markers desired, so may need to emit sequences such as `(?:\uFFFF\uE123)` meaning a match to marker #0x123 -- matching `\m{.}` may need to expand to `(?:\uFFFF[\uE000-\uEFFE])` +- Other emitters, such as `key`, `transform` will include the string `U+FFFF U+XXXX` where XXXX corresponds to the marker's 1-based number. +- Transforms will need to match against the marker or markers desired, so may need to emit sequences such as `(?:\uFFFF\u0123)` meaning a match to marker #0x0123 +- matching `\m{.}` may then expand to `(?:\uFFFF.)` ## Binary (.kmx plus) - The `vars.markers` is a pointer into the `list` section with a list (binary order) of the marker names -- Other strings will be in `U+FFFF U+E123` form etc. as if it was in the original text stream as such. +- Other strings will be in `U+FFFF U+0123` form etc. as if it was in the original text stream as such. ## Implementation (core) -- Core needs to recognize `U+FFFF …` sequences and convert them to markers in the context stream. -- For normal processing, Core does _not_ need to correlate the marker _number_ with a marker _id_, although this would be helpful for a debugging or tracing facility. I.e. `U+FFFF U+E123` corresponding to entry 0x123 in the `vars.markers` -> `list` table. +- Core needs to recognize `U+FFFF …` sequences and convert them to markers in the context stream, with `state->context().push_marker(marker_number)` +- For normal processing, Core does _not_ need to correlate the marker _number_ with a marker _id_, although this would be helpful for a debugging or tracing facility. I.e. `U+FFFF U+0123` corresponding to entry 0x0123 in the `vars.markers` -> `list` table. - Core needs to remove `U+FFFF …` sequences before they are passed to the OS. -- The default backspace processing needs to ignore `U+FFFF …` markers as it is deleting. + +- Transform processing needs to recognize these markers in the context stream and pass them to the transforms appropriately. +- User-defined backspace processing ` may specifically operate on backspaces in the context stream, just as with other transform processing. +- The default backspace processing needs to recognize these markers in the context stream and remove them as appropriate. [tr35-keyboards-markers]: https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-keyboards.md#markers