The compiler has always been very ambiguous on which characters were
accepted in group and store names, even to the point of accepting
things like comma in a store name, which would then make it impossible
to reference in an `index` statement!
This commit clarifies the allowable characters in an identifier. While
it would have been possible to use UAX#31 for this, that would have
extended the requirements for this change substantially, and may have
caused us more trouble with legacy keyboards. Given kmcmplib is
end-of-life (see epic/ng-compiler), I have chosen a lower friction
approach. There are certainly other characters that could be excluded,
but in general I have chosen to exclude only those that will definitely
be problematic.
The set of allowable characters for deadkeys has actually been expanded
in this release to match the store and group name rules.
It is expected that there may be some impacted keyboards, but addressing
this change will be relatively straightforward, so I consider this to be
an acceptable back-compatibility trade-off, see
https://github.com/keymanapp/keyman/wiki/Principles-of-Keyman-Code-Changes#4-source-backward-compatibility-keyboard-model-and-package-source-file-formats-should-be-backward-compatibleFixes: #14604
Test-bot: skip
Build-bot: skip build:developer
If `nul` is used in the output part of a rule, then text or
text-emitting statements cannot also be included in the output of that
rule. This was not enforced in earlier versions of Keyman.
Adds a unit test for the enhanced validation.
Fixes: #13455
Relates-to: keymanapp/keyboards#3379
Add validity checks to kmcmplib to verify that offsets provided to
`context(n)` are in range for the context, and do not point to
non-character elements (`nul` and `if`) in the context string.
Fixes: #13301Fixes: #13302
Relates-to: #13299
Relates-to: #13276
Adds 'verbose' and 'debug' log levels and corresponding error severity
levels. This means the internal error severity values have changed to
match - 'info' is now 2 instead of 0.
Fixes: #12975
* Remove compiler message definitions from kmcmplib
* Add parameterization to compiler message structures
* Translate parameters for existing parameterized messages (except for
`ERROR_InvalidToken`, which requires a bigger refactor of
`GetXStringImpl()` and many friends)
* Add columnNumber to message structures (not yet used in kmc-kmn)
* Add filename to message structures (not yet used in kmcmplib)
* Rename `INFO_Info` to `INFO_MinimumCoreEngineVersion` and
`INFO_MinimumEngineVersion` to `INFO_MinimumWebEngineVersion`
Relates-to: #10866
Add a unit test to verify that kmn_compiler_errors.h and
kmn-compiler-messages.ts have identical definitions for all errors in
kmcmplib. Then remove mismatched (and unused) messages identified by
this test.
Renames kmcmplib error codes to match those in kmc-kmn, and updates the
bitmasks accordingly. Adds a KmnCompilerMessages namespace so a trivial
enum can be declared without risk of clashes with other ERROR_ or
similar messages.
These two codes are not true error or message codes, but are used as
return values extensively throughout kmcmplib. The codes are never
passed to message callbacks. As such, I have renamed them to reflect
usage, but opted not to change their values as there would be some risk
in doing so (e.g. use of `!v` vs `v == CERR_None`).
While it does not cause any problems to have a index() store that is
longer than its corresponding any() store, it often indicates a mistake,
as the trailing characters in the store are ignored. Thus, adding as a
hint (which can be disabled via message suppression) rather than as a
warning or error.
Example code:
store(abc) 'abc'
store(defg) 'defg'
any(abc) + 'x' > index(defg, 1) c generates HINT_IndexStoreLong
Fixes: #10666
Currently, keys must be a UTF-16 code unit or a virtual key. Non-BMP
characters are unsupported. Technically, there is space available in the
.kmx `COMP_KEY` structure to accommodate UTF-32 codepoints, from
kmx_file.h:
KMX_WORD_unaligned Key;
KMX_WORD_unaligned _reserved;
However, the utility of this is almost nil, as it is very unlikely we
will encounter base keyboards (for mnemonic layouts) that generate any
characters outside the BMP, so there is little value in adding support
for this at this time.
The compiler will generate an error if this is encountered.
Fixes: #11643
Fixes#11092.
Addresses buffer boundary tests for four cases, so a fatal error is not
returned to the user:
* character range too long (U+1234 .. U+2468)
* extended string too long ('abcde...xxxxx')
* outs too long (store(foo) .... outs(bar))
* virtual key expansion too long ([K_A] .. [K_Z] ...)
See #11136 for additional work arising.
Fixes#10678.
Some messages have been reduced to ERROR instead of FATAL, as they are
not internal compiler errors.
Where fatal messages are raised, the message will now be reported
through correctly to Sentry.
Fixes#10059.
Use of the unsupported and undocumented virtual key output, that doesn't
work in recent Keyman versions, at all, now results in a build warning.
Only a warning, because it did kinda work in old versions of Keyman.
Fixes#7880.
Virtual keys have never been allowed in context. This should be an error
in a keyboard. Note: virtual keys in output are officially unsupported
but still kinda sorta a little bit work in Windows -- but I doubt they
will ever be officially supported.
Also includes small side journey to tidy up names and constants for two
other tests (error_duplicate_group and error_duplicate_store).
Fixes#9296.
208A in kmcmplib is the awkwardly-named CWARN_Info. We now have an
'info' severity in kmc, so this tidies that up (we can keep the same
flag in kmcmplib for now).
kmcmplib no longer has any filesystem access, so it cannot verify if a
referenced filename in a source file has the same case as the actual
filename on disk (a risk when moving projects between platforms). So
I opted to move this to the `loadFile` callback in kmc, which is the
only place where filesystem is actually accessed, and added
corresponding unit test.
Small additional fixes here:
1. Move from `Buffer` to `Uint8Array` in all kmc-* modules, so that we
remove that barrier to running on web.
2. Use `callbacks.loadFile` instead of `callbacks.fs.readFileSync`, so
that we can be sure to run the filename consistency check.
3. Fixed kps parser silently swallowing xml errors on load.
4. Added silent mode to NodeCompilerCallbacks so we could cleanly test
the new filename consistency hint.
5. Noted a location where we still have NodeJS deps in kmc-ldml.
Consolidates the two copies of comperr.h (they were identical) and
moves to a common folder. Updates all references to comperr.h, except
for one github commit ref.
