Add a KMX+ file reader and tests, including a round-trip builder:reader
test.
Move basic-17.txt, basic-19.txt, and basic.xml to common test resources
folder so they can be used by unit tests in common and developer.
Test-bot: skip
This replaces and enables the commented `k_000_null_keyboard` test which
didn't work because keys that are not on any layer don't produce output.
This instead defines a minimal keyboard with just two keys and then tests
typing a key that is on the keyboard followed by a key not on the keyboard.
Test-bot: skip
Instead of using an iterator to loop over the context items, incrementing
a counter on each iteration, and then finally removing the calculated
number of context items from the list, this change loops through the list
and looks at the last context item, removing it if necessary.
Follows: #15596
Test-bot: skip
Production code uses a list of context items, so this change modifies
the tests to also use a list instead of a vector to more closely match
production code.
Addresses code review comment.
Test-bot: skip
This change replaces the reverse iterator loop that holds a stale iterator
across `pop_back()` calls with a pattern that directly accesses
`context.back()` on each iteration. This avoids undefined behavior from
iterator invalidation when mutating the list or vector.
While so far the previous code didn't show problems, it might still access
released memory depending on the implementation. The documentation for
`pop_back()` says "References and iterators to the erased element are
invalidated", so the previous implementation was clearly wrong.
Test-bot: skip
When normalizing, we need to stop processing on an NFC boundary, not an
NFD boundary, to support normalizations such as in Bengali, where
appending `U+09D7` to a context of `U+0995 U+09C7` should result in
`U+0995 U+09CC`.
The specification is unclear on this; see https://unicode-org.atlassian.net/browse/CLDR-19218
This also updates the ldml keyboard unit test suite to support running
in full NFC mode (used in all Engine implementations) as well retaining
the NFD mode (now only used by the debugger).
Side note: the Bengali normalization failure case was picked up by the
improvements to the unit test suite, proving once again that good tests
are so valuable.
Fixes: #15491Fixes: #15505
Follows: #15488
Relates-to: CLDR-19218
Ensure that when a single backspace decomposes the last NFC character in
the app context, the remainder of the 'cluster' is preserved, matching
the implication of the CLDR keyboard specification.
This addresses the behavior in #15487 where the cached context became
out of sync with the app context after deleting an entire NFC cluster
such as ê, which caused a loop ending up with the entire context being
deleted, at which point the loop exited with a fail-safe.
Note that the LDML keyboard tests (ldml.cpp) do not currently exercise
the normalization code; this is a gap that should be addressed to ensure
that we are testing final application behavior.
Fixes: #15487
Add a GitHub action to test Keyman Core on Windows / ARM64. We do not
have a TeamCity ARM64 agent for this. There are a number of config
challenges in making this work as many deps do not yet support ARM64;
however the changes are isolated to the .yml.
Test-bot: skip
With VS2022 the limitations for the -hostarch flag not being available
on arm environment and the -arch only having x86 and x64 has been
removed. There is now single command that works for 3 builds on either
host architecture.
Fixes: 15239
Test-bot: skip
Implement the conversion from .kvk into KMX+ format, and embed the
result into the .kmx file. This change does not yet support &displaymap.
Move some virtual key modifier state helpers out of
visual-keyboard-compiler.ts and into shared module
modifier-key-constants.ts.
This change also flags the need to update the package compiler to
rewrite the font metadata into the KMX+ tables.
Test-bot: skip
Add a host check for build.bat and run the architecture tests
that match the host. for the meson builds from build.h
for now just skip the arm64 tests.
The disp and layr sections have new v19 layouts, to support the Keyman
OSK requirements for epic/embed-osk-in-kmx. This supports writing the
new versions of these sections in Developer and loading them into Core,
doing transformations where necessary so that Core always works with v19
structures after load.
This does not implement the transformations required to support the OSK
APIs; that will be implemented in a follow-up PR. Nor is support for
writing the OSK data from source .kvks and .keyman-touch-layout
supported; this is just the scaffolding for supporting the structures in
the KMX+ data.
Test-bot: skip
The name `form` better represents the semantic meaning of the
structure, and reduces confusion with the `list` section and other uses
of `list` in KMX+.
Test-bot: skip
Meson 1.1 changed the name of the options file from `meson_options.txt`
to `meson.options`. Currently we're stuck with Meson 1.0 as minimum
Meson version because we still support Ubuntu 22.04 Jammy, but on some
machines a newer version of meson gets installed which causes the
options to be ignored.
The first attempt to simply symlink or copy `meson_options.txt` to
`meson.options` failed because git on Windows doesn't support symlinks
by default, and Meson complains if it finds two options files (#15127).
Fortunately not having an options file is only a problem for Core where
we have one option in the options file that defaults to true. The solution
implemented in this change is to always pass the option to Meson.
Test-bot: skip
While not documented as such, it appears that the repository record is
required with trusted publishing of npm packages.
Test-bot: skip
Build-bot: skip
Update the build.bat to handle configuring the build
environment with vsdevcmd correctly for arm64 2019 doesn't
work very well so have to have pretty weird if statement.
also checked in some missed updates to function name change
Due to recent changes in NPM package publishing security requirements,
we have to move from TeamCity build to a GitHub Action to publish our
NPM packages, so we can take advantage of trusted publishing. This
change also consolidates and centralizes the npm publishing into
resources/build/ci/npm-publish.sh, which removes a lot of boilerplate
from each of the build.sh scripts, and ensures consistency.
Packages will be `npm pack`ed on PR and test builds, and published in
release builds.
Ref: https://docs.npmjs.com/trusted-publishers
Ref: https://github.blog/changelog/2025-09-29-strengthening-npm-security-important-changes-to-authentication-and-token-management/Fixes: #14963
Test-bot: skip
Build-bot: release:developer
Note: #11909 has some additional future cleanup we could do, but these
code paths are not currently accessible, so in the interest of moving
forward, am leaving them for now.
Fixes: #11909
Test-bot: skip
* Address cross-platform compiler warnings around shadowed pointers
and non-virtual destructors.
* Refactor the offset calculations for sub-section data to reduce
repetition both within each calculation, and the patterns of the
calculations themselves. This mostly resolves possibility of typo
errors when mapping the section data in, e.g. using the wrong count,
as each variable and type is only referenced once, and compiler will
catch most discrepancies - except for count vs type. This is also much
easier to read and verify in code review, I hope!
This substantial refactor reorganizes the header data for sections in
the Core LDML processor. The change was substantial because of
assumptions made about the binary layout of sections. In order to make
the code easier to maintain, safer, and more resilient to future
changes, I opted to make a consistent helper for each section, and copy
header data so that it could be transparently reused.
The principal changes are:
1. Support the version field in the COMP_KMXPLUS_HEADER struct, and
split it into COMP_KMXPLUS_HEADER_17 and COMP_KMXPLUS_HEADER_19
versions.
2. Establish a corresponding COMP_KMXPLUS_XXXX_Helper clas for each
section.
3. Refactor the majority of rawData accesses into using helper
functions, which reduces direct pointer manipulation and adds extra
boundary checks.
A special-case exists for BKSP - it is identical to TRAN, except for
its section ident. In order to avoid a complicated pattern for handling
it, I have special-cased it in one place, adding an overloaded
`get_section_from_sect` function for COMP_KMXPLUS_BKSP.
An opportunity exists to refactor a bit further - reduce direct access
to the binary data (through the COMP_KMXPLUS_XXXX structs, and instead
promote use of the COMP_KMXPLUS_XXXX_Helper classes). This indirection
would reduce duplication of data access and make it cleaner when we
start doing multi-version code.
Next commit will add unit tests for v19 format files.
Add support for header.version to KMX+ files, with differentiation
between v17 and v19 in the compiler, and associated tests. This does not
yet make the target version accessible to authors using `kmc`, but the
interfaces from `kmc-ldml` are available.
A v17 KMX+ file will have:
* comp_header.version = VERSION_17
* initial 'sect' section in KMX+ data
* no sect.header.version field in any section
A v19+ KMX+ file will have:
* comp_header.version = VERSION_19
* initial 'sec2' section in KMX+ data
* sect.header.version field for all sections
* each section 4 bytes larger to accommodate
* 'sec2' must have sect.header.version = KMXPlusVersion.Version19
* currently all other sections have sect.header.version = Version17
Removes a number of unused functions from the file, renames remaining
functions to match our builder script prefix patterns, removes unused
imports from various build scripts (e.g. Android).
Fixes: #14453
Consolidates the node-related script functions into node.inc.sh, as part
of cleaning up the build scripts and making them easier to maintain into
the future.
Fixes: #14447
Clarifies the confusing builder.inc.sh / build-utils.sh distinction by
giving the scripts more appropriate names. Most build scripts should use
builder-full.inc.sh; some helper scripts can use builder-basic.inc.sh.
Documented in resources/build/README.md.
Renames:
* resources/build/builder.inc.sh to resources/build/builder-full.inc.sh
* resources/build/build-utils.sh to resources/build/builder-basic.inc.sh
Other changes:
* Moves Android-specific functions out of builder-basic.inc.sh and into
android/build.sh.
* Renames functions in builder-basic.inc.sh
More functions may be moved from builder-basic.inc.sh into utils.inc.sh
or other scripts in the future.
Fixes: #14065
Build-bot: build all
Test-bot: skip
