Deploy kmc as an ES module, forced now by top-level await that we use in
the sentry load.
Moved NodeCompilerCallbacks to same util folder as schemas are found in,
to avoid rewriting schema loading with imports in this PR. It belongs
better there than messages/ anyway.
Distributed files are now .mjs. Removed --enable-source-maps from
launchers as logging will be managed with sentry anyway.
Fixes#9113.
Adds sentry reporting, plus helper functions, for kmc. Includes the
now-standard `-sentry-client-test-exception` command line parameter,
with `kmcmplib` and `event` sub-options available.
Uploads kmc* sourcemaps to sentry on a release build.
On Windows, honours the registry setting for error reporting. Also
includes a command line option `--[no-]error-reporting` (must be used
prior to initial command, e.g. `kmc --no-error-reporting build ...`).
Fixes#9110.
Also, `!=` has higher precedence than `??`.
This returns a boolean result:
```
(platform.font?.toLowerCase() ?? '') != FTouchLayoutFont
```
But `!=` has higher precedence than `??` so this returns the font name:
```
platform.font?.toLowerCase() ?? '' != FTouchLayoutFont
```
Fixes#9100.
The `compilerWarningsAsErrors` option can be passed in as a command-line
option or in the project options. Command-line option, if present, takes
precedence over the project option. If neither is set, then the value is
`false`.
Handling of this is within kmc, rather than individual modules.
Fixes#9233.
Adds --color and --no-color switches to kmc command line, with default
being determined by console mode.
Also ensures color initialization happens just once in
NodeCompilerCallbacks and moves constants to top of file.
Fixes#8795.
Relates to #9090.
Cleans up and makes consistent compiler messages in kmc, and related
unit tests.
Adds support for colorized messages.
After installing chalk for ansi coloring in kmc, @types/node needed to
be brought in sync with the Typescript version, per the suggestion of
checking against `npm dist-tags @types/node`,found in a discussion at
https://github.com/DefinitelyTyped/DefinitelyTyped/discussions/64262#discussioncomment-4905069
Additional tidying up includes:
* Deprecating a set of functions in compiler-interfaces, bringing them
together under a new CompilerError class.
Separating CompilerErrorSeverity and CompilerErrorMask enums.
* Making CompilerError.formatEvent (formerly compilerEventFormat) print
messages in the same format as NodeCompilerCallbacks (sans coloring).
* Adding a wrapper class for CompilerCallbacks that manages the filename
reporting, used currently exclusively by kmc itself.
Fixes#9208.
Fixes#9209.
Fixes#9203.
Fixes#9198.
The changes here overlap in kmp-compiler.ts. Addresses several issues
with the package compiler:
* refactors the reading of the metadata from keyboards to make use of it
when refreshing the keyboard metadata in kmp.json (#9208). This is the
bulk of the changes.
* Fixes case on some fields in the kps file format (#9209)
* Removes code which emitted strings table to kmp.json, as these were
only ever used for package installer executables (#9203)
* Cleans up code which emitted Start Menu items to make it match spec
(#9198)
Fixes#9183.
* CSS: blank line required at EOF if not present, but only if the CSS
file was not empty!
* Embedded HTML needs a blank line at EOF if not present
* `\r` is converted to `\\n` and `\n` is converted to ` `, to match
kmcomp
Fixes#9181.
Two issues:
* First use of a `call` statement had an incorrect index (misuse of
`push` return value)
* The .call_js files were incorrectly prepended with the basename of the
keyboard
- remove async SectionCompiler.init() and all callers
- the usetparser lives on the DependencySections, initialized by kmc-ldml (and by the test harnesses)
for: #7377
- update docs on UnicodeSet
- uset parser: fix/clarify that the wasm interface takes a bufferSize, but the ts interface takes a range count. This was muddled before.
For: #7377
Fixes#9154.
Improves performance in both kmcmplib and in kmc-kmn/kmw-compiler. After
these changes are applied, vietnamese_telex keyboard builds in around 6
seconds on my machine (down from over 2 minutes). This is certainly only
the surface of performance improvements we could make.
Addresses excessive memory reallocations in kmcmplib, by reallocating
store and key arrays in large chunks of 100 items rather than
reallocating on each new item added. This resulted in a 250x speed-up on
functions such as `AddStore` in WASM build.
Some careful modulus arithmetic was needed for resizing the key array,
which can be grown in larger increments.
Resolved excessive zeroing out of destination buffer in u16ncpy, which
was being called with an 8kb destination buffer on every line of the
file.
In the kmw side, vast majority of the performance cost was in copying of
buffers while loading a .kmx into memory, rather than creating views
into the memory. Essentially replaced this pattern:
```
return this.rString.fromBuffer(source.slice(offset));
```
with:
```
const data = new Uint8Array(source.buffer, source.byteOffset + offset);
return this.rString.fromBuffer(data);
```
Fixes#9145.
* `WARN_KeyboardVersionsDoNotMatch` is now only raised when
`FollowKeyboardVersion` is set.
* `WARN_KeyboardVersionsDoNotMatchPackageVersion` has been removed,
because it did not really make sense; if 'FollowKeyboardVersion' is
set, it could not be raised, and otherwise, the author may wish to
have separate keyboard + package versions anyway.
* Note that the 0x0013 compiler message allocation for
`WARN_KeyboardVersionsDoNotMatchPackageVersion` has been left alone;
as this was only used in pre-release, we can probably re-task it in
the future.
Fixes#9146.
For model packages, 304016 (ERROR_ModelMustHaveAtLeastOneLanguage)
remains as an error (this is a new error for kmc, kmcomp did not
validate model metadata). For keyboard packages, a new warning is
introduced to match the existing kmcomp behaviour, 20401B
(WARN_KeyboardShouldHaveAtLeastOneLanguage).
Fixes#9150.
Legacy .kmn keyboards can refer to a binary .kvk file. kmc needs to be
able to load these as well.
Note: there was a short period of time in which .kvk files were either
source or binary format. We moved to .kvk always being binary, and .kvks
always being source (xml), and so IMO we don't need to include support
for mismatched file formats.
Fixes#9140.
Fixes#9148.
Keyman Developer 9.0 .kpj files included a lot of additional state
metadata. We need a schema which validates these files, as they are
otherwise valid to load (we'll never save them any more). Rather than
add all the extra metadata to what is otherwise a fairly clean schema,
we'll provide a legacy .schema.json.
In the future, we may be able to merge these schemas, as we move towards
the .kpj 2.0 format which doesn't list files. Ideally, all three formats
(Keyman Developer 9.0 schema, call it legacy, 1.0 schema for Keyman
Developer 10.0+ which has Options and Files listed, 2.0 schema for
capturing just project settings for a folder) will be supported by a
single schema file.
- update element string constructors to support UnicodeString and the segmented array from ElementParser
- update compiler machinery to pass USetParser in the DependencySections
- test cases
#7377
Fixes#9111.
We have existing packages which have only .js in them, for touch-only
keyboards (mostly legacy but still...), so we need to support the freaky
Javascript regex search which we did in the past for extracting
metadata. While this is not necessarily going to work with hand-crafted
Javascript keyboards, it should work with all kmc- and kmcomp-generated
keyboards, so it will suffice to support these legacy packages.
In the future, we will be giving a hint when a package includes a .js
but not a .kmx, gradually upgrading this to a warning and finally an
error as we attempt to phase out .js-based keyboards in preference for
.kmx keyboards.
Running kmc on a set of keyboards would sporadically fail after some
time. It turns out this was related to memory allocation, specifically,
resizing the WASM module's memory buffer caused `TypedArray` views into
the memory to be reset!
By default, when a `Uint8Array` is created from an `ArrayBuffer` (e.g.
`Module.HEAP8.buffer`), it is a dynamic view into that buffer. This
module buffer can be dynamically reallocated at any time, which can
happen when allocating memory in WASM code (so the change will look
_really_ weird in a stack trace). Thus, to ensure we don't trip over
ourselves, we need to copy the buffer. Fortunately, creating a
`Uint8Array` from a `Uint8Array` copies the data, and is pretty quick.
