This change adds the mocha-teamcity-reporter which outputs the running
tests in a special format that TeamCity can interpret. This allows TC
to show which tests run as well as the number of tests run.
Also refactor some of the build scripts to use functions.
Note that we had to explicitly add `--check-coverage=false` (which is the
default) to get successful builds when we run on TC.
The Description field should be required for published keyboards and
models, so this change means kmc-keyboard-info and kmc-model-info will
report an error if it is missing.
Relates-to: keymanapp/keyboards#3037
Relates-to: keymanapp/lexical-models#262
Fixes: #12202
With the full move to ES Modules, we no longer need to include
`references` in tsconfig.json, as we can rely on package.json and
build.sh dependency management. Note however that `tsc -b` may not work
to build dependencies -- they need to be built using `build.sh`, which
calculates which dependencies need building.
kmc-keyboard-info was missing a dependency link to kmc-package in
build.sh, correcting this at the same time.
Relates-to: #12027
Add verification of email address formats to kmc-package,
kmc-keyboard-info, and kmc-model-info. Add corresponding unit tests for
kmc-package and kmc-keyboard-info (kmc-model-info unit tests will be a
later addition). Uses AJV's email validation regex, as we rely on AJV
for schema compliance tests.
Fixes: #11362
Fixes: KEYMAN-DEVELOPER-1XE
Use a new flag `--npm-publish` in conjunction with `publish` action,
so that the default will always be to `npm pack` if the new flag is not
specified. This flag is also guarded in the actual npm publish code to
ensure that it can only run in the appropriate CI alpha/beta/stable
environment, and not in local or test.
This then also removes the separate `pack` action.
Also removes legacy boilerplate from a number of build scripts.
Note: there is a bit of potential confusion about the difference between
/resources/builder.inc.sh (the full implementation for builder scripts),
and /resources/build/builder.inc.sh (the source script that builder
scripts should always use).
This allows us to make assumptions that will always be true for builder
scripts that may not be true for other scripts, such as setting base
folder.
Fixes#11324.
* Always `cd "$THIS_SCRIPT_PATH"`
* Remove unnecessary `cd` from all build.sh
* Remove unnecessary `set -eu` from all build.sh (and `# set -x`)
* Replace old build-utils.sh incantation in a few build.sh scripts
Establishes keyman-urls.ts as a single place for any links to external
websites.
I searched all .ts files in Keyman Developer source and replaced
references to websites with function calls here. Note that the names of
individual functions within this class can be easily changed as needed,
as the hard part was the consolidation.
Realtes to #10207.
Adds a new `kmc message` command which prints detailed information about
any or all compiler messages, in JSON, text, or markdown formats.
Markdown format is emitted to file in a structured way to be deployed to
the help site automatically (coming in a subsequent commit).
Moved the ERROR_Message line above the Error_Message line for every
message in the compiler message files, as it makes the grouping clearer
once we start adding message details. For example, see the start of the
KmnCompilerMessages class.
Also wraps potentially-undefined parameters to the messages with a new
function `def`, shorthand for `CompilerMessageDef`, which converts the
parameter to '<param>' if it is undefined, which is helpful for
documentation.
Relates to #10207.
In order to include message detail in messages, it is helpful to split
out the exception messages which have a separate parameter, so that
parameter at the same position can be the message detail.
Introduces the CompilerMessageSpecWithException function which takes an
exceptionVar parameter to achieve this differentiation.
Relates to #10207.
Adds the `api` action to build.sh for each affected module. The `api`
action for each module will generate the api metadata files, and then
the markdown files are generated by /developer/build.sh api.
Note that developer-utils, kmc, and kmcmplib have no api documentation
at this time (and kmcmplib is not Typescript so would need separate
tooling anyway).
Relates to #10207.
Adds @microsoft/api-extractor and @microsoft/api-documenter NPM
packages for validating and generating API documentation from Typescript
modules.
Adds basic configuration for each module that we will be publishing at
this time.
api-extractor's output will be emitted to /developer/docs/api/etc/, for
future release comparisons. A baseline output will be committed
separately.
The build outputs will be in /developer/build/docs and the intermediate
files will be in /developer/build/api.
Note that at this time, we will not enable the checks and validations.
That is a task for alpha rather than beta. The primary intent at this
time is to select a solid, well-supported toolchain for generating the
API documentation for kmc.
Relates to #10150. Makes the various Messages classes in kmc modules
accessible to other modules. Some file renames involved to reduce
confusion.
No functional changes.
Fixes kmc-analyze metadata.
See #10254 for a related future fix.
Relates to #9473.
Refactors the public API of ModelInfoCompiler to meet KeymanCompiler,
including moving file write responsibilities into the class itself.
`sources` has become a property of `options`, which is perhaps a slight
violation of the intent of the `init()` function being a one-time call,
as it means we'll need to instantiate an new class for each file we
compile. Given the use-case for the model_info compiler is almost
exclusively internal, I think this is acceptable. Tell me if you think
otherwise!
Given we've just run a build, this (a) seems unnecessary, and (b) seems
to go wrong in some circumstances anyway, e.g.
https://build.palaso.org/buildConfiguration/Keyman_Developer_Release/421326?buildTab=log&focusLine=19936&linesState=18839&logView=flowAware:
```
07:01:29 > @keymanapp/kmc@17.0.205-alpha prepublishOnly
07:01:29 > npm run build
07:01:29
07:01:31
07:01:31 > @keymanapp/kmc@17.0.205-alpha build
07:01:31 > tsc -b
07:01:31
07:01:35 ../../../common/web/types/build/src/kmx/kmx.d.ts(1,1): error TS1036: Statements are not allowed in ambient contexts.
07:01:35 ../../../common/web/types/build/src/kmx/kmx.d.ts(1,2): error TS1345: An expression of type 'void' cannot be tested for truthiness.
...
```
The new --for-publishing flag allows us to enforce additional
requirements for keyboards and lexical models that are to be published
to the keymanapp repositories. This flag will always be switched on in
the repository builds. This flag overrides the 'skipMetadata' option
which is a project-level option.
There is a bit of a delicate balance of requirements here:
1. We want to be able to automatically verify keyboards and package
licenses if they are in the repository.
2. We do not want to force license checks on privately built keyboards
and models.
3. We want to provide pathways for users to check locally before
submitting to the repository.
4. If possible, we want to be able to build the .keyboard_info and
.model_info files locally, but this should be up to the user for
local builds.
For now, the only additional check that this flag provides is to verify
that the license is MIT, which was an unchecked requirement for the
repositories in the past.
Future checks can be added for file layout, additional required
metadata files.
Anticipate adding this as a tool to Keyman Developer IDE -- a
'pre-publish' check -- in the future.
I missed adding this as a dependency for @keymanapp/kmc-package. because
node has a very lenient dependency search, it found xml2js from other
@keymanapp/common-types, so never caused errors locally.
Adds support for calculating isRTL and license fields in .model_info
compiler.
Also:
* establishes @keymanapp/developer-utils shared module
* moves license validation into @keymanapp/developer-utils
* refactors kmc-model-info to a class and general cleanup
Removes all references to source .keyboard_info and .model_info files in
the kmc compiler and types. Updates the model info compiler to support
building purely from model sources (isRTL, license fields are TODO).
Relates to #9351.
Adds support for building a .keyboard_info file without having source
.keyboard_info file:
* Constructs a default source .keyboard_info in memory
* Hints if license is missing
* If LICENSE.md is present, verifies it matches the MIT license text
* Adds project option to turn on or off metadata generation. This will
default to False for version 1.0 projects, and to True for version 2.0
projects. This means that the keyboard repository will need a PR to
enable metadata generation for existing projects in the repository,
but this is important to avoid breaking builds for existing projects
that are not in the repository.
* Turns on additional c8 coverage for kmc projects
kmc already reported unhandled exceptions, but any handled fatal errors
were captured and only reported to the user. It is better to report
these to Sentry as these are still unexpected.
I have refactored all the fatal exception messages in various kmc
modules to use a common mechanism, keeping all the Sentry integration in
kmc, now passing exception data up in the `CompilerEvent.exceptionVar`
property.
* I took the opportunity to rename messages.ts to
infrastructureMessages.ts
* @types/chai was missing which gave intellisense errors in vscode
* normal exit of kmc now provides an opportunity for error reports to
Sentry to be finalized
* Added a unit test for fatal errors in kmc
* Added a manual test pathway with `SENTRY_CLIENT_TEST_BUILD_EXCEPTION`
env var to trip the build fatal error mechanism and verify that it
looks ok; the following shows test runs demonstrate how fatal build
errors are reported:
```
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
This error has been automatically reported to the Keyman team.
Identifier: 6f0fca1a26694c22b03f02b2463d39c5
Application: Keyman Developer
Reported at: https://sentry.io/organizations/keyman/projects/keyman-developer/events/6f0fca1a26694c22b03f02b2463d39c5/
mcdurdin@THARK MINGW64 /c/Projects/keyman/app/developer/src/kmc (chore/developer/report-fatal-compiler-errors-to-sentry)
$ SENTRY_CLIENT_TEST_BUILD_EXCEPTION=1 node . --no-error-reporting build
fatal KM05001: Unexpected exception: Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
Call stack:
Error: Test exception from SENTRY_CLIENT_TEST_BUILD_EXCEPTION
at build (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:78:19)
at Command.<anonymous> (file:///C:/Projects/keyman/app/developer/src/kmc/build/src/commands/build.js:66:24)
at Command.listener [as _actionHandler] (C:\Projects\keyman\app\node_modules\commander\lib\command.js:482:17)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:65
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1283:27)
at C:\Projects\keyman\app\node_modules\commander\lib\command.js:1081:27
at Command._chainOrCall (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1177:12)
at Command._dispatchSubcommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1077:23)
at Command._parseCommand (C:\Projects\keyman\app\node_modules\commander\lib\command.js:1248:19)
```
