This comes out of a design philosophy review on what we include when we
embed OSK data into KMX.
We will now avoid embedding font name into the OSK (and hence .kmx)
altogether, and leave that metadata to the packaging data. Reasons:
1. The font information is specified in the .kps, so we have to do a
patchup on the .kmx during packaging if we want to embed the info
into the OSK.
2. The referenced font must be supplied separately anyway (via .kmp,
@font-face, or system supplied, etc), so including the font facename
in the keyboard is not really all that helpful.
3. Philosophically, the font is really a presentation level factor
(aside from displaymap considerations). Keeping it together with
future theming and styling choices, rather than the key layout data,
seems appropriate.
4. This makes fewer places where font data is referenced -- in fact, to
just one place: in the .kps/.kmp for LDML keyboards, which is great.
This also simplifies some aspects of the embed-osk-in-kmx work, removing
the need to patch the .kmx after the build, and eliminates the smelly
kmx-plus-osk-token.ts file.
A corresponding change has been made to the design document referenced
in #14857.
Test-bot: skip
Instead of using an iterator to loop over the context items, incrementing
a counter on each iteration, and then finally removing the calculated
number of context items from the list, this change loops through the list
and looks at the last context item, removing it if necessary.
Follows: #15596
Test-bot: skip
This change replaces the reverse iterator loop that holds a stale iterator
across `pop_back()` calls with a pattern that directly accesses
`context.back()` on each iteration. This avoids undefined behavior from
iterator invalidation when mutating the list or vector.
While so far the previous code didn't show problems, it might still access
released memory depending on the implementation. The documentation for
`pop_back()` says "References and iterators to the erased element are
invalidated", so the previous implementation was clearly wrong.
Test-bot: skip
When normalizing, we need to stop processing on an NFC boundary, not an
NFD boundary, to support normalizations such as in Bengali, where
appending `U+09D7` to a context of `U+0995 U+09C7` should result in
`U+0995 U+09CC`.
The specification is unclear on this; see https://unicode-org.atlassian.net/browse/CLDR-19218
This also updates the ldml keyboard unit test suite to support running
in full NFC mode (used in all Engine implementations) as well retaining
the NFD mode (now only used by the debugger).
Side note: the Bengali normalization failure case was picked up by the
improvements to the unit test suite, proving once again that good tests
are so valuable.
Fixes: #15491Fixes: #15505
Follows: #15488
Relates-to: CLDR-19218
Ensure that when a single backspace decomposes the last NFC character in
the app context, the remainder of the 'cluster' is preserved, matching
the implication of the CLDR keyboard specification.
This addresses the behavior in #15487 where the cached context became
out of sync with the app context after deleting an entire NFC cluster
such as ê, which caused a loop ending up with the entire context being
deleted, at which point the loop exited with a fail-safe.
Note that the LDML keyboard tests (ldml.cpp) do not currently exercise
the normalization code; this is a gap that should be addressed to ensure
that we are testing final application behavior.
Fixes: #15487
The disp and layr sections have new v19 layouts, to support the Keyman
OSK requirements for epic/embed-osk-in-kmx. This supports writing the
new versions of these sections in Developer and loading them into Core,
doing transformations where necessary so that Core always works with v19
structures after load.
This does not implement the transformations required to support the OSK
APIs; that will be implemented in a follow-up PR. Nor is support for
writing the OSK data from source .kvks and .keyman-touch-layout
supported; this is just the scaffolding for supporting the structures in
the KMX+ data.
Test-bot: skip
The name `form` better represents the semantic meaning of the
structure, and reduces confusion with the `list` section and other uses
of `list` in KMX+.
Test-bot: skip
Note: #11909 has some additional future cleanup we could do, but these
code paths are not currently accessible, so in the interest of moving
forward, am leaving them for now.
Fixes: #11909
Test-bot: skip
* Address cross-platform compiler warnings around shadowed pointers
and non-virtual destructors.
* Refactor the offset calculations for sub-section data to reduce
repetition both within each calculation, and the patterns of the
calculations themselves. This mostly resolves possibility of typo
errors when mapping the section data in, e.g. using the wrong count,
as each variable and type is only referenced once, and compiler will
catch most discrepancies - except for count vs type. This is also much
easier to read and verify in code review, I hope!
This substantial refactor reorganizes the header data for sections in
the Core LDML processor. The change was substantial because of
assumptions made about the binary layout of sections. In order to make
the code easier to maintain, safer, and more resilient to future
changes, I opted to make a consistent helper for each section, and copy
header data so that it could be transparently reused.
The principal changes are:
1. Support the version field in the COMP_KMXPLUS_HEADER struct, and
split it into COMP_KMXPLUS_HEADER_17 and COMP_KMXPLUS_HEADER_19
versions.
2. Establish a corresponding COMP_KMXPLUS_XXXX_Helper clas for each
section.
3. Refactor the majority of rawData accesses into using helper
functions, which reduces direct pointer manipulation and adds extra
boundary checks.
A special-case exists for BKSP - it is identical to TRAN, except for
its section ident. In order to avoid a complicated pattern for handling
it, I have special-cased it in one place, adding an overloaded
`get_section_from_sect` function for COMP_KMXPLUS_BKSP.
An opportunity exists to refactor a bit further - reduce direct access
to the binary data (through the COMP_KMXPLUS_XXXX structs, and instead
promote use of the COMP_KMXPLUS_XXXX_Helper classes). This indirection
would reduce duplication of data access and make it cleaner when we
start doing multi-version code.
Next commit will add unit tests for v19 format files.
- improve an output string
- catch BMP noncharacters besides U+FFFF (H/T @mcdurdin - this was the whole point of the PR) plus test
Fixes: #9446
Co-authored-by: Marc Durdin <marc@durdin.net>
- improve how validation works - a missing section does not mean an invalid section. distingush these.
- propagate errors for invalid sections
- update documentation of required sections
Fixes: #9446
- turn off some asserts- makes untestable
(there are asserts at 'higher levels' such as loading the entire kmx+)
- add a test with a synthesized COMP_KMXPLUS_STRS - a valid and an invalid one
Fixes: #9446
Two separate bugs addressed, with `index()` references and with
`context()` references -- both have the same root cause, of not taking
`nul` at the start of the context into account (as `nul` is not included
in the `m_miniContext` member, being a non-character). We already fixed
this issue for `if()` quite a long time ago, and some of the same
patterns can be with `m_miniContextIfLen` for example.
Fixes: #13304Fixes: #13316
- yes, expand 'other' to all possible combinations
- use ALT and CTRL instead of RALT,LALT and RCTRL,LCTRL in the key list (reduce expansions up to 4x)
Fixes: #12298
This change disables logging at compile time to work around #12661.
Logging can be enabled in the debugger, or by re-compiling with
`g_debug_KeymanLog` set to TRUE.
Related: #12661
Cherry-pick: #12674
- split keyboard loading into loading KMX file into blob and then
loading the keyboard processor from the blob.
- deprecate `km_core_keyboard_load`
- move file access next to deprecated method. This is now the only place
that loads a file in Core; unit tests have some more places that
load files.
- introduce GTest and add unit tests for loading from blob
Cherry-picked from `epic/web-core` branch.
Cherry-Pick-Commit: 1deaa323ad
Cherry-Pick-Commit: 59019cc8b7
Cherry-Pick-Commit: bc46458368
Cherry-Pick-Commit: d06aa29956
Cherry-Pick-Commit: 1c88166f6e
Cherry-Pick-Commit: 069cd21ecd
Cherry-Pick-Commit: 052ae2ec35
Cherry-Pick-Commit: 11a2a3ba3a
Part-of: #11293
Part-of: #8093
- move kmxplus processing into the base LdmlTestSource class
- add a function to traverse the layer list looking for keys to add
- The @@keylist keyword only has one example from each modifier set
Fixes: #12298
