The url module is a node module. We need to move responsibility for
resolving the path of the LDML XML <import> statements out of
common/web/types, and into the ultimate consumer, so it's now surfaced
as an option, along with a helper constant that reports the
import.meta.url-relative base path of the standard imports that are
compiled into common/web/types.
This hopefully means we can use this module in both browser and node
contexts without trouble.
kmcmplib no longer has any filesystem access, so it cannot verify if a
referenced filename in a source file has the same case as the actual
filename on disk (a risk when moving projects between platforms). So
I opted to move this to the `loadFile` callback in kmc, which is the
only place where filesystem is actually accessed, and added
corresponding unit test.
Small additional fixes here:
1. Move from `Buffer` to `Uint8Array` in all kmc-* modules, so that we
remove that barrier to running on web.
2. Use `callbacks.loadFile` instead of `callbacks.fs.readFileSync`, so
that we can be sure to run the filename consistency check.
3. Fixed kps parser silently swallowing xml errors on load.
4. Added silent mode to NodeCompilerCallbacks so we could cleanly test
the new filename consistency hint.
5. Noted a location where we still have NodeJS deps in kmc-ldml.
Relates to #8493.
* Removes kmcmplib calls from kmcmpdll (now that we have kmc)
* Removes old kmcmp_CompileKeyboardFile and
kmcmp_CompileKeyboardFileToBuffer functions in preference for a much
cleaner kmcmp_CompileKeyboard function
* Removes json validation helper from kmcmplib (we'll use js-native
json schema validation instead)
This change means that we no longer need to keep compfile.h consistent
between kmcmplib and kmcmpdll. This will simplify upcoming refactoring
of kmcmplib.
kmcmplib no longer writes files, but it does still read them. The next
refactor will move file load responsibility into the caller.
FILE_KEYBOARD is a structure used in multiple libraries and languages.
It is passed around between kmcmplib, kmcmpdll, and kmcomp, with three
separate definitions (kmcmplib/compfile.h, kmcmpdll/compfile.h, and
kmcomp/compile.pas). These duplicate definitions must be maintained for
now because of type issues between the legacy kmcmpdll and kmcmplib,
which means that the definitions must be kept in sync.
Because FILE_KEYBOARD_EXTRA has a C++ class in it, it will be
initialized in CompileKeyboardHandle rather than in the caller, so
FILE_KEYBOARD.extra is now a pointer rather than a struct.
When original kmcomp and kmcmpdll are eventually removed, these
duplications will disappear and some of the memory management and
lifecycle will become easier to manage.
Refactor the `extractKeyboardVersionFromKmx` function into a new class
and add extra version checking. This caused a bit of a cascade of test
failures due to some of the 'invalid' fixtures actually being invalid
in multiple ways, so the fixtures have been corrected to only be wrong
in a single way -- the way they are supposed to be broken. This means
additional fixture files in the 'invalid' folder.
Now updates keyboard version metadata for all keyboards in the package.
Upgrades `WARN_FollowKeyboardVersionButNoKeyboards` to
`ERROR_FollowKeyboardVersionButNoKeyboards`, as this leads to invalid
package metadata on build.
Renames `ERROR_KeyboardFileNotFound` to
`ERROR_KeyboardContentFileNotFound` to better reflect that no `<File>`
in the package is found to match a given `<Keyboard>` entry.
Adds `ERROR_KeyboardFileNotFound` when a referenced .kmx does not exist.
Adds and updates corresponding unit tests.
developer:
- update wasm machinery in kmc-kmn to be more self contained
- improve exception situation in wasm functions
common:
- compilerErrorFormatCode() for formatting the raw code such as for tests
- compilerExceptionToString() for formatting exceptions in messages
for: #7234
Fixes#8644.
This is a signficant cleanup and refactor of code that was accessing the
fs or path modules in Node directly. Given we want to be able to run
these modules on web as well in the future, it is important that we
avoid that.
This also redesigns the CompilerCallback interfaces to provide Node-like
interfaces for path and fs, with a minimal set of functions. The aim
here is to provide a surface that will hopefully match a future
web-based component such as path-browserify without significant
retooling of the kmc-* modules.
Enables linting for common/web/types and turns on the
eslintNoNodeImports checks for that module and kmc-package.
Note: Buffer is still in use in a few places also, so we'll need to
refactor that in a future commit.
Cleans up majority of remaining kmc-* modules to use CompilerEvent
messages instead of console.log and friends, adding
InfrastructureMessages class for kmc CLI messages.
Consolidates schema loading functions into a single `loadSchema`
function in the CompilerCallbacks interface and implementations.
Moves responsibility for instantiating NodeCompilerCallbacks out of
each individual BuildActivity and into higher level modules.
Fixes return value for failing builds.
Replaces the existing error reporting in kmc-model with CompilerEvent
style messages. Some of the existing error reporting was kinda nice, but
consistency trumps pretty here.
One downside at present is the global filename and line variables in the
model-compiler-errors.ts unit. This is fragile, and should probably be
refactored further in the future.
Where it made sense, I converted various throws into ModelCompilerError
instances. If it appeared to be an internal error, I left it as-is, but
the external API interfaces wrap all unknown exceptions and convert them
into a compiler error message, so these APIs (compileModel and
loadFromFilename) now have a guarantee that they will never throw.
At this point, all of the error messages in the model compiler are
unrecoverable -- they will halt the compile entirely. However, the
pathway is there to support recoverable errors through the normal
callbacks.reportMessage mechanisms.
The message-counting mechanism is currently missing from this, as it
belongs in the API consumer, rather than in the low-level reporting
infrastructure. This means that models with large numbers of warnings
are going to generate scads of messages for now.
Note that the _users_ of CompilerCallbacks are subject to change in the
future, once we have all the compilers using the callbacks.
Consolidates the two copies of comperr.h (they were identical) and
moves to a common folder. Updates all references to comperr.h, except
for one github commit ref.
Fixes#8443.
Will cherry-pick to stable-16.0.
The regex for matching error messages has an optional section for
the file/line detail (line 54):
```regex
^(?:(.+) \((\d+)\): )?<snip>
```
The code that parses the results did not account for this being
optional, which would cause a crash if the error message did not include
this information.
This fixes two issues:
1. `COMP_KEY` structure had padding that was not always zeroed out.
Made this explicit.
2. Rule sorting could be inconsistent when re-sorting rules that
use `+ any() > ...`, which are expanded at compile time into multiple
rules, and which ended up with identical sort keys due to lack of
precision. This relates to #8381 but only adds differentiation for
the otherwise ambiguous rule sort keys, and does not fix that issue,
which requires further investigation.
Relates to #4324 (but slightly orthogonal.)
We wanted some message classes to be handled in the lexical model editor
more cleanly, including drill-down to line of error. A few changes
required to make this work well:
1. Parse error messages from kmlmc. This is not perfect but works while
the two projects are kept in sync, which they always are for Keyman
Developer. Note that at this stage, .ts warnings are not captured in
this parser, as they are generated by tsc.
2. Drill-down in Wordlist Editor Frame to find line of error
3. Model editor reports ownership of .tsv files so they can be loaded
4. In case of model editor not open (e.g. building from project view),
the TSV standalone editor was not displaying the frame, so it never
actually worked.
5. If the text editor had never loaded, then FindError was effectively
a no-op; adds code to seek to error line after page load finishes.
6. Ensures that if we attempt to seek an error in a sub-file owned by
an editor (e.g. .tsv owned by .model.ts, .kvks owned by .kmn), that
the parent editor will be focused first. Does not verify all paths
here, just the tsv one.
Fixes#7216.
The warning message 0x209A 'The rule will never be matched because its
key code is never fired.' was being generated multiple times for a
single line because the `JavaScript_Key` function it is generated by is
used for various purposes.
This PR keeps a cache of reported key rules to ensure that the message
is reported only once for a given key rule, and also improves the
reporting to clarify which specific key is unreachable, which makes it
easier to diagnose when using `any(k)` style messages, for example:
```
lao_phonetic.kmn (237): Warning: 209A The rule will never be matched for key 'ñ' because its key code is never fired.
```
This also reduces the warning to a hint, as this should not be a
blocking issue for a keyboard, rather just a place the keyboard author
can tidy up.
If `-no-compiler-version` is specified, then we skip the embedding of
the `TSS_COMPILEDVERSION` and `TSS_KEYMANCOPYRIGHT` stores. This is
intended for use with regression test builds of keyboards when we want
to verify changes to the compiler.
This patch introduces the new export `SetCompilerOptions` in kmcmpdll,
which should make it easier to include new options like this in future
versions.
Fixes the release build where we do `nmake symbols` for developer/src,
which was failing on calls to these two sets of components. (In
windows/src, the build structure is different due to an additional layer
of folders, so we end up with no `nmake symbols` call on the common
components).
Note this also drags TextFileFormat.pas and KMDActionInterfaces.pas into
the package so have moved those into general and components folders
respectively, alongside KMDActions.pas and updated corresponding
project includes.
