This starts the refactor of the .kmn compiler interfaces for WASM,
moving more file writes into typescript and out of the compiler. Adds a
cleaner binding and structures for passing compile metadata back to
typescript for additional build steps (e.g. kvk, js builds).
Will rebase the Typescript-based KMW compiler onto this so it can start
to benefit from the refactoring.
* Fixes a minor issue with loading a .kvks file with an empty flags set.
* Adds a writeFileSync method to the compiler callback interfaces.
* Adds an e2e compiler test to verify that .kmx and .kvk are both
correctly compiled.
developer:
- update wasm machinery in kmc-kmn to be more self contained
- improve exception situation in wasm functions
common:
- compilerErrorFormatCode() for formatting the raw code such as for tests
- compilerExceptionToString() for formatting exceptions in messages
for: #7234
Fixes#8644.
This is a signficant cleanup and refactor of code that was accessing the
fs or path modules in Node directly. Given we want to be able to run
these modules on web as well in the future, it is important that we
avoid that.
This also redesigns the CompilerCallback interfaces to provide Node-like
interfaces for path and fs, with a minimal set of functions. The aim
here is to provide a surface that will hopefully match a future
web-based component such as path-browserify without significant
retooling of the kmc-* modules.
Enables linting for common/web/types and turns on the
eslintNoNodeImports checks for that module and kmc-package.
Note: Buffer is still in use in a few places also, so we'll need to
refactor that in a future commit.
Cleans up majority of remaining kmc-* modules to use CompilerEvent
messages instead of console.log and friends, adding
InfrastructureMessages class for kmc CLI messages.
Consolidates schema loading functions into a single `loadSchema`
function in the CompilerCallbacks interface and implementations.
Moves responsibility for instantiating NodeCompilerCallbacks out of
each individual BuildActivity and into higher level modules.
Fixes return value for failing builds.
Read a .kmx file into memory in Typescript. Currently only reads stores
from the file, as these are the most common things needed for other
processes. Will add support for reading other metadata and rules in
future when required.
Replaces the existing error reporting in kmc-model with CompilerEvent
style messages. Some of the existing error reporting was kinda nice, but
consistency trumps pretty here.
One downside at present is the global filename and line variables in the
model-compiler-errors.ts unit. This is fragile, and should probably be
refactored further in the future.
Where it made sense, I converted various throws into ModelCompilerError
instances. If it appeared to be an internal error, I left it as-is, but
the external API interfaces wrap all unknown exceptions and convert them
into a compiler error message, so these APIs (compileModel and
loadFromFilename) now have a guarantee that they will never throw.
At this point, all of the error messages in the model compiler are
unrecoverable -- they will halt the compile entirely. However, the
pathway is there to support recoverable errors through the normal
callbacks.reportMessage mechanisms.
The message-counting mechanism is currently missing from this, as it
belongs in the API consumer, rather than in the low-level reporting
infrastructure. This means that models with large numbers of warnings
are going to generate scads of messages for now.
Note that the _users_ of CompilerCallbacks are subject to change in the
future, once we have all the compilers using the callbacks.
Consolidates the two copies of comperr.h (they were identical) and
moves to a common folder. Updates all references to comperr.h, except
for one github commit ref.
Relates to #8150.
Legacy .kmp compiler would transform xml-format .kvk files into a binary
.kvk file; now we want that to remain the responsibility of the keyboard
compiler, so we'll warn the few users who are still doing this.
Moves to using the `reportMessage` callback for reporting all errors in
kmc-kmn.
This required some patching to common/web/types in order to get the new
constants in place and add support for line numbers to `CompilerEvent`,
so at the same time I refactored `compilerErrorSeverityName` into the
module.
After the refactor, coverage tests were failing because they included
the test/ folder, so cleaned that up at the same time.
The build/ folder for most packages was missing due to being listed in
.gitignore.
Also, @keymanapp/ldml-keyboard-constants package was not being published
despite being a dependency of @keymanapp/kmc-keyboard.
Instead of relying on npm's use of .gitignore / .npmignore (which has
seriously wonky behaviours), we list the files that should be included
in package.json.
NPM Wonky behaviours: it seems that .npmignore causes .gitignore in the
same folder to be ignored. But, higher level .gitignore files still
affect the files included in the package, which makes
specification-by-exclusion painful (negating exclusions, etc). We do not
recommend use of .npmignore anywhere for this reason.
Fixes#8586.
All npm publish actions in the repository are now run as part of the
Developer release build, hosted at present by kmc, until we setup a
higher-level build script to do it.
Adds a 'pack' action which mirrors 'publish' for all npm packages.
Removes unnecessary dev dependencies of models-templates and
models-wordbreakers from kmc-model.
Web Release build CI step has been updated to check for presence of
.build-builder.
Improves build script performance by:
* using built-ins wherever possible (e.g. string splitting)
* eliminating redundant code
* using absolute (to $KEYMAN_ROOT) rather than relative paths to avoid realpath
* removing unnecessary `npm run` calls
BEFORE | AFTER
-------------------------------|----------------------
time ./core/build.sh --help |
real 0m2.116s | real 0m0.874s
user 0m0.578s | user 0m0.198s
sys 0m0.984s | sys 0m0.289s
-------------------------------|----------------------
time ./web/build.sh --help |
real 0m3.523s | real 0m0.757s
user 0m1.166s | user 0m0.320s
sys 0m2.273s | sys 0m0.455s
-------------------------------|----------------------
time ./web/build.sh -d |
real 1m34.750s | real 0m59.974s
user 0m9.721s | user 0m6.284s
sys 0m19.652s | sys 0m13.202s
@keymanapp-test-bot skip
Relates to #2761.
kmc now supports passing in a folder and building all Keyman source
files found in the folder, even if a .kpj is not present in the folder.
If a .kpj file exists, `KeymanDeveloperProject/Options/Version` will be
checked to determine behaviour. If this field is missing or "1.0", then
the classical behaviour persists.
If the Version field is "2.0", the `Files` key is ignored, and .kmn,
.xml, .model.ts, and .kps files under `sourcePath` are enumerated and
added to the in-memory project for build.
If a .kpj is not present in the folder, a virtual project is constructed
in memory with the following defaults:
* `version`: `2.0`
* `buildPath`: `$PROJECTPATH/build`
* `sourcePath`: `$PROJECTPATH/source`
* `compilerWarningsAsErrors`: `false`
* `warnDeprecatedCode`: `true`
* `checkFilenameConventions`: `true`
It is still considered an error to include both a keyboard and a model
in the same file, but a version 2.0 project will not include the field
determining its type in `Options/ProjectType`; this will instead be
checked during build.
The call to `kmc` can reference a folder or a .kpj file. The .kpj file
must have the same basename as the folder name (e.g.
`khmer_angkor/khmer_angkor.kpj`).
WARNING: this does not add support for 2.0 projects to Keyman Developer;
it is likely that opening a version 2.0 project in Developer will
currently corrupt it.
WARNING: it is not anticipated that kmcomp.exe will be updated to
support version 2.0 projects -- rather kmcomp.exe will be removed before
release of v17, and replaced with kmc.
Adds support for building .kpj projects. Required minor tweak to kmcomp
to allow it to build both .kmx and .js from .kmn command line.
Currently, calls kmcomp.exe to build .kmn files, until we support a
cross-platform kmc-kmn C++ project.
