Fixes#9154.
Improves performance in both kmcmplib and in kmc-kmn/kmw-compiler. After
these changes are applied, vietnamese_telex keyboard builds in around 6
seconds on my machine (down from over 2 minutes). This is certainly only
the surface of performance improvements we could make.
Addresses excessive memory reallocations in kmcmplib, by reallocating
store and key arrays in large chunks of 100 items rather than
reallocating on each new item added. This resulted in a 250x speed-up on
functions such as `AddStore` in WASM build.
Some careful modulus arithmetic was needed for resizing the key array,
which can be grown in larger increments.
Resolved excessive zeroing out of destination buffer in u16ncpy, which
was being called with an 8kb destination buffer on every line of the
file.
In the kmw side, vast majority of the performance cost was in copying of
buffers while loading a .kmx into memory, rather than creating views
into the memory. Essentially replaced this pattern:
```
return this.rString.fromBuffer(source.slice(offset));
```
with:
```
const data = new Uint8Array(source.buffer, source.byteOffset + offset);
return this.rString.fromBuffer(data);
```
Fixes#9145.
* `WARN_KeyboardVersionsDoNotMatch` is now only raised when
`FollowKeyboardVersion` is set.
* `WARN_KeyboardVersionsDoNotMatchPackageVersion` has been removed,
because it did not really make sense; if 'FollowKeyboardVersion' is
set, it could not be raised, and otherwise, the author may wish to
have separate keyboard + package versions anyway.
* Note that the 0x0013 compiler message allocation for
`WARN_KeyboardVersionsDoNotMatchPackageVersion` has been left alone;
as this was only used in pre-release, we can probably re-task it in
the future.
Fixes#9146.
For model packages, 304016 (ERROR_ModelMustHaveAtLeastOneLanguage)
remains as an error (this is a new error for kmc, kmcomp did not
validate model metadata). For keyboard packages, a new warning is
introduced to match the existing kmcomp behaviour, 20401B
(WARN_KeyboardShouldHaveAtLeastOneLanguage).
Fixes#9150.
Legacy .kmn keyboards can refer to a binary .kvk file. kmc needs to be
able to load these as well.
Note: there was a short period of time in which .kvk files were either
source or binary format. We moved to .kvk always being binary, and .kvks
always being source (xml), and so IMO we don't need to include support
for mismatched file formats.
Fixes#9140.
Fixes#9148.
Keyman Developer 9.0 .kpj files included a lot of additional state
metadata. We need a schema which validates these files, as they are
otherwise valid to load (we'll never save them any more). Rather than
add all the extra metadata to what is otherwise a fairly clean schema,
we'll provide a legacy .schema.json.
In the future, we may be able to merge these schemas, as we move towards
the .kpj 2.0 format which doesn't list files. Ideally, all three formats
(Keyman Developer 9.0 schema, call it legacy, 1.0 schema for Keyman
Developer 10.0+ which has Options and Files listed, 2.0 schema for
capturing just project settings for a folder) will be supported by a
single schema file.
Fixes#9111.
We have existing packages which have only .js in them, for touch-only
keyboards (mostly legacy but still...), so we need to support the freaky
Javascript regex search which we did in the past for extracting
metadata. While this is not necessarily going to work with hand-crafted
Javascript keyboards, it should work with all kmc- and kmcomp-generated
keyboards, so it will suffice to support these legacy packages.
In the future, we will be giving a hint when a package includes a .js
but not a .kmx, gradually upgrading this to a warning and finally an
error as we attempt to phase out .js-based keyboards in preference for
.kmx keyboards.
Running kmc on a set of keyboards would sporadically fail after some
time. It turns out this was related to memory allocation, specifically,
resizing the WASM module's memory buffer caused `TypedArray` views into
the memory to be reset!
By default, when a `Uint8Array` is created from an `ArrayBuffer` (e.g.
`Module.HEAP8.buffer`), it is a dynamic view into that buffer. This
module buffer can be dynamically reallocated at any time, which can
happen when allocating memory in WASM code (so the change will look
_really_ weird in a stack trace). Thus, to ensure we don't trip over
ourselves, we need to copy the buffer. Fortunately, creating a
`Uint8Array` from a `Uint8Array` copies the data, and is pretty quick.
Picked this up while running `-fsanitize=address` for trying to track
down another bug.
This fixes a buffer overrun (normally invisible) in kmcmplib, where
`xstrchr` was being called, which takes a string as its second
parameter, but being passed a single character. This was incorrect
behaviour for two reasons:
1. Passing a single character doesn't have null termination (big bug!)
2. We were not wanting xstring semantics on the token being parsed here
Replaced with `u16chr`, which does do what we want. Note the
casting because `u16chr` only has a `const` version of its input/output
at present.
Removed `xstrchr` because it is not used anywhere else in kmcmplib.
The kmcmplib compiler source used `delete` instead of `delete[]` in a
number of locations where the corresponding allocation was an array.
This doesn't appear to matter on arrays of primitives on most platforms
that we are targeting, but it _is_ undefined behaviour so we should
correct it.
https://stackoverflow.com/a/2425749/1836776
The tests for layout validity in #9087 were incomplete; first, they
didn't fail the build if a key had invalid identifier, and second, there
were some constants missing from the KeyIdType test (imperfect
translation from the Delphi code in this case).
Also removes leftover commented code, and changes an enum to string enum
type in order to get constant names for free for an error message.
Error_TouchLayoutIdentifierRequires15 was split into its own error
message because it was overloaded with CERR_TouchLayoutInvalidIdentifier
previously but had a different message.
Adds support for readonly groups and corresponding unit test. Again
required more metadata from kmcmplib; this metadata is now fairly
straightforward to patch in without side-effects, which is encouraging.
Adds support for option stores to kmc-kmn/kmw-compiler. This required
adding the relevant store metadata into the output from kmcmplib, and
so I also tidied up the corresponding WASM interfaces slightly more in
the process.
Added unit test for the options store.
While testing this, ran into a second bug with the way I ported a
constant from Delphi to Typescript in constants.ts, so fixed that and
added a corresponding unit test. Small steps, but good ones.
Reorganizes the data structures passed out of kmcmplib via WASM to make
it easier to work with the extra metadata and reduce the number of
places we have to touch when we add extra metadata fields.
Preparation for supporting the metadata that the kmw compiler requires.
