Relates to #9948.
While the schema validation does check the version already, it gives a
fairly obtuse message, and we can do better.
Found a secondary issue with a iterator type guard needed in
kpj-file-reader when preparing unit test, which was probably masked in
other tests by not verifying all messages generated, so updated all
infrastructure message tests to verify the full set of messages
generated.
Fixes#9610.
If a build fails when "treat warnings as errors" is on, but there are
only warnings and no error messages, it can be confusing for the
developer. Adds the following `INFO_WarningsHaveFailedBuild` message:
`The build failed because option "treat warnings as errors" is enabled
and there are one or more warnings.`
Relates to #9351.
Adds support for building a .keyboard_info file without having source
.keyboard_info file:
* Constructs a default source .keyboard_info in memory
* Hints if license is missing
* If LICENSE.md is present, verifies it matches the MIT license text
* Adds project option to turn on or off metadata generation. This will
default to False for version 1.0 projects, and to True for version 2.0
projects. This means that the keyboard repository will need a PR to
enable metadata generation for existing projects in the repository,
but this is important to avoid breaking builds for existing projects
that are not in the repository.
* Turns on additional c8 coverage for kmc projects
kmc already reported unhandled exceptions, but any handled fatal errors
were captured and only reported to the user. It is better to report
these to Sentry as these are still unexpected.
I have refactored all the fatal exception messages in various kmc
modules to use a common mechanism, keeping all the Sentry integration in
kmc, now passing exception data up in the `CompilerEvent.exceptionVar`
property.
* I took the opportunity to rename messages.ts to
infrastructureMessages.ts
* @types/chai was missing which gave intellisense errors in vscode
* normal exit of kmc now provides an opportunity for error reports to
Sentry to be finalized
* Added a unit test for fatal errors in kmc
* Added a manual test pathway with `SENTRY_CLIENT_TEST_BUILD_EXCEPTION`
env var to trip the build fatal error mechanism and verify that it
looks ok; the following shows test runs demonstrate how fatal build
errors are reported:
```
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
This error has been automatically reported to the Keyman team.
Identifier: 6f0fca1a26694c22b03f02b2463d39c5
Application: Keyman Developer
Reported at: https://sentry.io/organizations/keyman/projects/keyman-developer/events/6f0fca1a26694c22b03f02b2463d39c5/
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --no-error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
```
Fixes missing components of the kmc-keyboard-info compiler and
integration into kmc. Lots of cleanup making sure that the compiler
actually generates expected output against the keyboards repository.
Adds ability to build .model_info and .keyboard_info files to kmc. Note
that .keyboard_info building is currently a stub as the .keyboard_info
compiler has not yet been translated to TypeScript, which will come in a
subsequent PR.
Note that .model_info and .keyboard_info builds depend on the project
loader, as they reference both keyboard and package source files when
constructing the metadata files.
Also:
* Fixed a deep reference to KeymanDeveloperProjectFile.
* Split the project loader into a utility unit.
* Marked the .kpj fileType field as deprecated, as it is easily
extrapolated from the filename, with the aim to remove it entirely
from version 2.0 projects.
@keymanapp-test-bot skip
Fixes#9162.
Adds support for `@files.txt` parameters in kmc. Files listed in this
file, separated by line breaks, will be added to the input file list,
relative to the path of files.txt.
The file can include comments in lines starting with '#' and ignores
whitespace before and after each filename.
Deploy kmc as an ES module, forced now by top-level await that we use in
the sentry load.
Moved NodeCompilerCallbacks to same util folder as schemas are found in,
to avoid rewriting schema loading with imports in this PR. It belongs
better there than messages/ anyway.
Distributed files are now .mjs. Removed --enable-source-maps from
launchers as logging will be managed with sentry anyway.
Fixes#8998.
kmc-kmn messages were broken down into sub-namespaces, but the range
overlapped the reserved namespace mask range. Fixed the message values,
added a new test to the verifyCompilerMessagesObject function to verify
that messages don't creep outside their namespace, and defined new
masks (in CompilerErrorSeverity enum... hmm) to help.
I opted to remove the leading `0` in the messages for kmc-kmn as I had
to correct a number of them anyway, but not the messages in remaining
units at this time (it's not an error, but just slightly misleading as
we only have 12 bits, not 16 to play with).
It is safe to reassign these messages as they were only assigned in 17.0
alpha.
Part 1 of the `CompilerOptions` reorg. kmc now uses `CompilerOptions`
throughout, and `CompilerBaseOptions` maps to the set of options
available to all activities.
Tidies up silent vs quiet vs verbose by adding logLevel as an option.
* Notes that we need to consolidate all the various CompilerOptions
interfaces before they get too much further out of hand.
* Cleans up a couple of other minor TODO items.
kmcmplib no longer has any filesystem access, so it cannot verify if a
referenced filename in a source file has the same case as the actual
filename on disk (a risk when moving projects between platforms). So
I opted to move this to the `loadFile` callback in kmc, which is the
only place where filesystem is actually accessed, and added
corresponding unit test.
Small additional fixes here:
1. Move from `Buffer` to `Uint8Array` in all kmc-* modules, so that we
remove that barrier to running on web.
2. Use `callbacks.loadFile` instead of `callbacks.fs.readFileSync`, so
that we can be sure to run the filename consistency check.
3. Fixed kps parser silently swallowing xml errors on load.
4. Added silent mode to NodeCompilerCallbacks so we could cleanly test
the new filename consistency hint.
5. Noted a location where we still have NodeJS deps in kmc-ldml.
Downgrades `Warn_KeyboardFileHasNoKeyboardVersion` to
`Info_KeyboardFileHasNoKeyboardVersion`, because this is not an error,
or even something wrong necessarily; it's just something it's good to be
aware of. Updates unit test for kmc to cater for the extra message.
Relates to #2761.
kmc now supports passing in a folder and building all Keyman source
files found in the folder, even if a .kpj is not present in the folder.
If a .kpj file exists, `KeymanDeveloperProject/Options/Version` will be
checked to determine behaviour. If this field is missing or "1.0", then
the classical behaviour persists.
If the Version field is "2.0", the `Files` key is ignored, and .kmn,
.xml, .model.ts, and .kps files under `sourcePath` are enumerated and
added to the in-memory project for build.
If a .kpj is not present in the folder, a virtual project is constructed
in memory with the following defaults:
* `version`: `2.0`
* `buildPath`: `$PROJECTPATH/build`
* `sourcePath`: `$PROJECTPATH/source`
* `compilerWarningsAsErrors`: `false`
* `warnDeprecatedCode`: `true`
* `checkFilenameConventions`: `true`
It is still considered an error to include both a keyboard and a model
in the same file, but a version 2.0 project will not include the field
determining its type in `Options/ProjectType`; this will instead be
checked during build.
The call to `kmc` can reference a folder or a .kpj file. The .kpj file
must have the same basename as the folder name (e.g.
`khmer_angkor/khmer_angkor.kpj`).
WARNING: this does not add support for 2.0 projects to Keyman Developer;
it is likely that opening a version 2.0 project in Developer will
currently corrupt it.
WARNING: it is not anticipated that kmcomp.exe will be updated to
support version 2.0 projects -- rather kmcomp.exe will be removed before
release of v17, and replaced with kmc.
* Add a bunch of tests for keys
* Add tests for version. (improves meta test coverage to 100%)
* A bit more consistency cleanup also of error message functions.
* Adds a compiler failure (FATAL) if compile fails after validate
passes.
* Overall test coverage now 93.5% lines. Biggest gap still kmx-builder.
All compiler units are >90%.
Fixes#7273.
Adds 4 kmx system stores to allow existing configuration tools to load
and process KMXPlus files without modification:
* `TSS_NAME` (first `<name>`)
* `TSS_COMPILEDVERSION` (only if `options.addCompilerVersion == true`)
* `TSS_KEYBOARDVERSION` (from `<version number>`)
* `TSS_TARGETS` (currently always 'desktop', will later include more
targets)
In order for `TSS_COMPILEDVERSION` to be handled correctly, needed to
plumb in the new `CompilerOptions` interface. This could be a separate
PR, but the changes are not huge, so opted to append the changes here.
Updates related tests and fixtures. Refactored some of the e2e helpers
into helpers/index.ts for use with the metadata-compiler tests.
Fixes#7235.
Encapsulates all strings in the in-memory compile so that we can binary
sort them for build. This means that finalization of strs and elem is
no longer needed -- it's part of setup of build_strs and build_elem --
so that is factored away. Renamed alloc_string and alloc_element_string.
It may be worth renaming some of the ELEM_STRING objects / interfaces as
currently they are a little confusing.
Fixes#7237.
Full clean, configure and build will be required for Core, Developer
after this change.
kmc is much less unwieldy a name than kmldmlc, and becomes our platform
for future compiler integration. Start of implementation for #5283 also.
