The Description field should be required for published keyboards and
models, so this change means kmc-keyboard-info and kmc-model-info will
report an error if it is missing.
Relates-to: keymanapp/keyboards#3037
Relates-to: keymanapp/lexical-models#262
Fixes: #12202
Establishes keyman-urls.ts as a single place for any links to external
websites.
I searched all .ts files in Keyman Developer source and replaced
references to websites with function calls here. Note that the names of
individual functions within this class can be easily changed as needed,
as the hard part was the consolidation.
Realtes to #10207.
Adds a new `kmc message` command which prints detailed information about
any or all compiler messages, in JSON, text, or markdown formats.
Markdown format is emitted to file in a structured way to be deployed to
the help site automatically (coming in a subsequent commit).
Moved the ERROR_Message line above the Error_Message line for every
message in the compiler message files, as it makes the grouping clearer
once we start adding message details. For example, see the start of the
KmnCompilerMessages class.
Also wraps potentially-undefined parameters to the messages with a new
function `def`, shorthand for `CompilerMessageDef`, which converts the
parameter to '<param>' if it is undefined, which is helpful for
documentation.
Relates to #10207.
In order to include message detail in messages, it is helpful to split
out the exception messages which have a separate parameter, so that
parameter at the same position can be the message detail.
Introduces the CompilerMessageSpecWithException function which takes an
exceptionVar parameter to achieve this differentiation.
Relates to #10207.
Adds the `api` action to build.sh for each affected module. The `api`
action for each module will generate the api metadata files, and then
the markdown files are generated by /developer/build.sh api.
Note that developer-utils, kmc, and kmcmplib have no api documentation
at this time (and kmcmplib is not Typescript so would need separate
tooling anyway).
Relates to #10207.
Adds @microsoft/api-extractor and @microsoft/api-documenter NPM
packages for validating and generating API documentation from Typescript
modules.
Adds basic configuration for each module that we will be publishing at
this time.
api-extractor's output will be emitted to /developer/docs/api/etc/, for
future release comparisons. A baseline output will be committed
separately.
The build outputs will be in /developer/build/docs and the intermediate
files will be in /developer/build/api.
Note that at this time, we will not enable the checks and validations.
That is a task for alpha rather than beta. The primary intent at this
time is to select a solid, well-supported toolchain for generating the
API documentation for kmc.
Relates to #10150. Makes the various Messages classes in kmc modules
accessible to other modules. Some file renames involved to reduce
confusion.
No functional changes.
Fixes kmc-analyze metadata.
See #10254 for a related future fix.
Relates to #9473.
Refactors the public API of ModelInfoCompiler to meet KeymanCompiler,
including moving file write responsibilities into the class itself.
`sources` has become a property of `options`, which is perhaps a slight
violation of the intent of the `init()` function being a one-time call,
as it means we'll need to instantiate an new class for each file we
compile. Given the use-case for the model_info compiler is almost
exclusively internal, I think this is acceptable. Tell me if you think
otherwise!
Given we've just run a build, this (a) seems unnecessary, and (b) seems
to go wrong in some circumstances anyway, e.g.
https://build.palaso.org/buildConfiguration/Keyman_Developer_Release/421326?buildTab=log&focusLine=19936&linesState=18839&logView=flowAware:
```
07:01:29 > @keymanapp/kmc@17.0.205-alpha prepublishOnly
07:01:29 > npm run build
07:01:29
07:01:31
07:01:31 > @keymanapp/kmc@17.0.205-alpha build
07:01:31 > tsc -b
07:01:31
07:01:35 ../../../common/web/types/build/src/kmx/kmx.d.ts(1,1): error TS1036: Statements are not allowed in ambient contexts.
07:01:35 ../../../common/web/types/build/src/kmx/kmx.d.ts(1,2): error TS1345: An expression of type 'void' cannot be tested for truthiness.
...
```
The new --for-publishing flag allows us to enforce additional
requirements for keyboards and lexical models that are to be published
to the keymanapp repositories. This flag will always be switched on in
the repository builds. This flag overrides the 'skipMetadata' option
which is a project-level option.
There is a bit of a delicate balance of requirements here:
1. We want to be able to automatically verify keyboards and package
licenses if they are in the repository.
2. We do not want to force license checks on privately built keyboards
and models.
3. We want to provide pathways for users to check locally before
submitting to the repository.
4. If possible, we want to be able to build the .keyboard_info and
.model_info files locally, but this should be up to the user for
local builds.
For now, the only additional check that this flag provides is to verify
that the license is MIT, which was an unchecked requirement for the
repositories in the past.
Future checks can be added for file layout, additional required
metadata files.
Anticipate adding this as a tool to Keyman Developer IDE -- a
'pre-publish' check -- in the future.
I missed adding this as a dependency for @keymanapp/kmc-package. because
node has a very lenient dependency search, it found xml2js from other
@keymanapp/common-types, so never caused errors locally.
Adds support for calculating isRTL and license fields in .model_info
compiler.
Also:
* establishes @keymanapp/developer-utils shared module
* moves license validation into @keymanapp/developer-utils
* refactors kmc-model-info to a class and general cleanup
Removes all references to source .keyboard_info and .model_info files in
the kmc compiler and types. Updates the model info compiler to support
building purely from model sources (isRTL, license fields are TODO).
Relates to #9351.
Adds support for building a .keyboard_info file without having source
.keyboard_info file:
* Constructs a default source .keyboard_info in memory
* Hints if license is missing
* If LICENSE.md is present, verifies it matches the MIT license text
* Adds project option to turn on or off metadata generation. This will
default to False for version 1.0 projects, and to True for version 2.0
projects. This means that the keyboard repository will need a PR to
enable metadata generation for existing projects in the repository,
but this is important to avoid breaking builds for existing projects
that are not in the repository.
* Turns on additional c8 coverage for kmc projects
kmc already reported unhandled exceptions, but any handled fatal errors
were captured and only reported to the user. It is better to report
these to Sentry as these are still unexpected.
I have refactored all the fatal exception messages in various kmc
modules to use a common mechanism, keeping all the Sentry integration in
kmc, now passing exception data up in the `CompilerEvent.exceptionVar`
property.
* I took the opportunity to rename messages.ts to
infrastructureMessages.ts
* @types/chai was missing which gave intellisense errors in vscode
* normal exit of kmc now provides an opportunity for error reports to
Sentry to be finalized
* Added a unit test for fatal errors in kmc
* Added a manual test pathway with `SENTRY_CLIENT_TEST_BUILD_EXCEPTION`
env var to trip the build fatal error mechanism and verify that it
looks ok; the following shows test runs demonstrate how fatal build
errors are reported:
```
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
This error has been automatically reported to the Keyman team.
Identifier: 6f0fca1a26694c22b03f02b2463d39c5
Application: Keyman Developer
Reported at: https://sentry.io/organizations/keyman/projects/keyman-developer/events/6f0fca1a26694c22b03f02b2463d39c5/
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --no-error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
```
Fixes missing components of the kmc-keyboard-info compiler and
integration into kmc. Lots of cleanup making sure that the compiler
actually generates expected output against the keyboards repository.
Fixes#9286.
Refactors the use of path and fs into the standard kmc callbacks pattern
and adds `callbacks.fileSize()`, which is needed for populating the
metadata file.
Turns on eslint test for node dependencies for kmc-model-info.
Fixes#8795.
Relates to #9090.
Cleans up and makes consistent compiler messages in kmc, and related
unit tests.
Adds support for colorized messages.
After installing chalk for ansi coloring in kmc, @types/node needed to
be brought in sync with the Typescript version, per the suggestion of
checking against `npm dist-tags @types/node`,found in a discussion at
https://github.com/DefinitelyTyped/DefinitelyTyped/discussions/64262#discussioncomment-4905069
Additional tidying up includes:
* Deprecating a set of functions in compiler-interfaces, bringing them
together under a new CompilerError class.
Separating CompilerErrorSeverity and CompilerErrorMask enums.
* Making CompilerError.formatEvent (formerly compilerEventFormat) print
messages in the same format as NodeCompilerCallbacks (sans coloring).
* Adding a wrapper class for CompilerCallbacks that manages the filename
reporting, used currently exclusively by kmc itself.
Enables eslint checks for eslintNoNodeImports for kmc-kmn. Also enables
linting for kmc-model-info, but does not enable eslintNoNodeImports on
it at this time.
Relates to #8959.
Adds eslint, effectively disabled, to infrastructure for all of kmc.
Note that eslintNoNodeImports.js will be used in the next commit, for
verifying #8644.
The build/ folder for most packages was missing due to being listed in
.gitignore.
Also, @keymanapp/ldml-keyboard-constants package was not being published
despite being a dependency of @keymanapp/kmc-keyboard.
Instead of relying on npm's use of .gitignore / .npmignore (which has
seriously wonky behaviours), we list the files that should be included
in package.json.
NPM Wonky behaviours: it seems that .npmignore causes .gitignore in the
same folder to be ignored. But, higher level .gitignore files still
affect the files included in the package, which makes
specification-by-exclusion painful (negating exclusions, etc). We do not
recommend use of .npmignore anywhere for this reason.
Fixes#8586.
All npm publish actions in the repository are now run as part of the
Developer release build, hosted at present by kmc, until we setup a
higher-level build script to do it.
Adds a 'pack' action which mirrors 'publish' for all npm packages.
Removes unnecessary dev dependencies of models-templates and
models-wordbreakers from kmc-model.
Web Release build CI step has been updated to check for presence of
.build-builder.
Improves build script performance by:
* using built-ins wherever possible (e.g. string splitting)
* eliminating redundant code
* using absolute (to $KEYMAN_ROOT) rather than relative paths to avoid realpath
* removing unnecessary `npm run` calls
BEFORE | AFTER
-------------------------------|----------------------
time ./core/build.sh --help |
real 0m2.116s | real 0m0.874s
user 0m0.578s | user 0m0.198s
sys 0m0.984s | sys 0m0.289s
-------------------------------|----------------------
time ./web/build.sh --help |
real 0m3.523s | real 0m0.757s
user 0m1.166s | user 0m0.320s
sys 0m2.273s | sys 0m0.455s
-------------------------------|----------------------
time ./web/build.sh -d |
real 1m34.750s | real 0m59.974s
user 0m9.721s | user 0m6.284s
sys 0m19.652s | sys 0m13.202s
@keymanapp-test-bot skip
