Relates to #10207.
Adds the `api` action to build.sh for each affected module. The `api`
action for each module will generate the api metadata files, and then
the markdown files are generated by /developer/build.sh api.
Note that developer-utils, kmc, and kmcmplib have no api documentation
at this time (and kmcmplib is not Typescript so would need separate
tooling anyway).
Fixes#10678.
Some messages have been reduced to ERROR instead of FATAL, as they are
not internal compiler errors.
Where fatal messages are raised, the message will now be reported
through correctly to Sentry.
Fixes#10059.
Use of the unsupported and undocumented virtual key output, that doesn't
work in recent Keyman versions, at all, now results in a build warning.
Only a warning, because it did kinda work in old versions of Keyman.
Fixes#7880.
Virtual keys have never been allowed in context. This should be an error
in a keyboard. Note: virtual keys in output are officially unsupported
but still kinda sorta a little bit work in Windows -- but I doubt they
will ever be officially supported.
Also includes small side journey to tidy up names and constants for two
other tests (error_duplicate_group and error_duplicate_store).
Fixes#9541.
Also adds version 16.0 support to the kmcmplib compiler constants, and
unit tests for both versions. Does not add any support for automatic
version feature detection, because that forces an inverted dependency on
the touch layout compilation phase (done in kmc-kmw), which would be a
significant refactor. This may be something we need to support in the
future.
Removes kmcomp.dpr, related source, compilekeymanweb.pas, and related
unit tests.
Cleans up a couple of bits and pieces around kmc in order to help tests
to pass, and updates some tests that had legacy code in them
(particularly missing &TARGETS stores).
Fixes#9113.
Adds sentry reporting, plus helper functions, for kmc. Includes the
now-standard `-sentry-client-test-exception` command line parameter,
with `kmcmplib` and `event` sub-options available.
Uploads kmc* sourcemaps to sentry on a release build.
On Windows, honours the registry setting for error reporting. Also
includes a command line option `--[no-]error-reporting` (must be used
prior to initial command, e.g. `kmc --no-error-reporting build ...`).
Fixes#9154.
Improves performance in both kmcmplib and in kmc-kmn/kmw-compiler. After
these changes are applied, vietnamese_telex keyboard builds in around 6
seconds on my machine (down from over 2 minutes). This is certainly only
the surface of performance improvements we could make.
Addresses excessive memory reallocations in kmcmplib, by reallocating
store and key arrays in large chunks of 100 items rather than
reallocating on each new item added. This resulted in a 250x speed-up on
functions such as `AddStore` in WASM build.
Some careful modulus arithmetic was needed for resizing the key array,
which can be grown in larger increments.
Resolved excessive zeroing out of destination buffer in u16ncpy, which
was being called with an 8kb destination buffer on every line of the
file.
In the kmw side, vast majority of the performance cost was in copying of
buffers while loading a .kmx into memory, rather than creating views
into the memory. Essentially replaced this pattern:
```
return this.rString.fromBuffer(source.slice(offset));
```
with:
```
const data = new Uint8Array(source.buffer, source.byteOffset + offset);
return this.rString.fromBuffer(data);
```
Picked this up while running `-fsanitize=address` for trying to track
down another bug.
This fixes a buffer overrun (normally invisible) in kmcmplib, where
`xstrchr` was being called, which takes a string as its second
parameter, but being passed a single character. This was incorrect
behaviour for two reasons:
1. Passing a single character doesn't have null termination (big bug!)
2. We were not wanting xstring semantics on the token being parsed here
Replaced with `u16chr`, which does do what we want. Note the
casting because `u16chr` only has a `const` version of its input/output
at present.
Removed `xstrchr` because it is not used anywhere else in kmcmplib.
The kmcmplib compiler source used `delete` instead of `delete[]` in a
number of locations where the corresponding allocation was an array.
This doesn't appear to matter on arrays of primitives on most platforms
that we are targeting, but it _is_ undefined behaviour so we should
correct it.
https://stackoverflow.com/a/2425749/1836776
Adds support for readonly groups and corresponding unit test. Again
required more metadata from kmcmplib; this metadata is now fairly
straightforward to patch in without side-effects, which is encouraging.
Adds support for option stores to kmc-kmn/kmw-compiler. This required
adding the relevant store metadata into the output from kmcmplib, and
so I also tidied up the corresponding WASM interfaces slightly more in
the process.
Added unit test for the options store.
While testing this, ran into a second bug with the way I ported a
constant from Delphi to Typescript in constants.ts, so fixed that and
added a corresponding unit test. Small steps, but good ones.
Reorganizes the data structures passed out of kmcmplib via WASM to make
it easier to work with the extra metadata and reduce the number of
places we have to touch when we add extra metadata fields.
Preparation for supporting the metadata that the kmw compiler requires.
Moves kmc-kmw to being a sub-component of kmc-kmn, as it can never be
independently instantiated anyway. Reorganized tests accordingly.
Note that c8 is currently disabled for kmw-compiler, until we add unit
tests for it.
kmc-kmn calls into kmc-kmw to build .js when it is needed, rather than
kmc-kmw depending on kmc-kmn. This means examining the `&targets` system store
to determine which files need to be generated.
This also implements the `&displayMap` rewrite for both .kvks and
.keyman-touch-layout as part of the KMW compiler.
I suspect that the next step is to place kmc-kmw into a subfolder of
kmc-kmn, given they are part of the same process, and there is no
practical way to separate out the .js generation from the .kmx
generation.
The &displayMap system store adds support for a mapping file that remaps
the On Screen Keyboard files -- .kvks and .keyman-touch-layout, at
compile time, to new ranges. This is specifically intended to resolve a
limitation of unattached marks and diacritics which display
inconsistently across platforms; this is described in more detail in
issue #9031.
Fixes#8955.
Adds a mapping table for CP1252 to USV so that we can compile very old
"ANSI"-encoded Keyman keyboard files with kmcmplib, to match kmcmpdll
support.
Fixes#8885.
Adds some optimization and error checking to the loadFile callback in
kmc-kmn. Also handles case of zero-byte bitmap file so we won't crash on
it.
kmcmplib no longer has any filesystem access, so it cannot verify if a
referenced filename in a source file has the same case as the actual
filename on disk (a risk when moving projects between platforms). So
I opted to move this to the `loadFile` callback in kmc, which is the
only place where filesystem is actually accessed, and added
corresponding unit test.
Small additional fixes here:
1. Move from `Buffer` to `Uint8Array` in all kmc-* modules, so that we
remove that barrier to running on web.
2. Use `callbacks.loadFile` instead of `callbacks.fs.readFileSync`, so
that we can be sure to run the filename consistency check.
3. Fixed kps parser silently swallowing xml errors on load.
4. Added silent mode to NodeCompilerCallbacks so we could cleanly test
the new filename consistency hint.
5. Noted a location where we still have NodeJS deps in kmc-ldml.
Fixes#8889.
No code changes, just moves WASM interfaces into
CompilerInterfacesWasm.cpp, and CompileKeyboardHandle is renamed to
CompileKeyboardBuffer and moved into its own source file.
* Moves filesystem access out of kmcmplib into kmc-kmn
* Adds filesystem access callback to kmcmplib unit tests
* Cleans up callback interface through wasm
* Adds unit tests for various file load scenarios
* Removes nodefs dependency from kmcmplib wasm build, and removes
corresponding path mappings which were previously required for wasm
builds; note that these are still present for the unit tests for
kmcmplib.
