Commit graph

1052 commits

Author SHA1 Message Date
Marc Durdin
cc0ddffd61 feat(developer): add vkey support
vkeyMaps table compiler, unit test, and fixes to json schema. Adds a
vkeyMaps entry to the basic e2e test.

us-virtual-keys.ts has a new mapping for CLDR VKey Enums, but I will
tweak this in an upcoming commit to use constants instead of hard-coded
values.
2022-09-05 10:14:10 +10:00
Marc Durdin
c08dbfbf9f feat(developer): add unit tests for 'name' compiler 2022-09-04 12:40:02 +10:00
Marc Durdin
b2b62733e6 refactor(developer): kmldmlc - rename CompilerErrors to CompilerMessages
Just wanted to rename this to make it clearer. Also added a prefix to
each message generator to indicate the type of message, as otherwise it
is not obvious within the compiler which messages are going to cause the
compiler to fail.

Note that the message generator function prefixes are Title Cased
whereas the message identifier constant prefixes are CAPS. I guess this
could be confusing but I think it works okay in practice.
2022-09-04 12:28:25 +10:00
Marc Durdin
2c4e3e109a feat(developer): add compiler hint for non-minimal loca
If a loca entry is either formatted wrongly (e.g. wrong case), or is
not minimal, then the compiler will now report a hint.
2022-09-04 12:21:15 +10:00
Marc Durdin
878287ef46 chore(developer): reorder functions for clarity 2022-09-04 06:20:07 +10:00
Marc Durdin
ea6bd8f09f chore(core): order sections according to binary format
Just reorders all work around the various sections to match the binary
format order -- that is, 'sect' first, then all other sections
alphabetically.

This also means the compiler now emits the sections in the expected
order.

No other changes to functionality.
2022-09-04 06:10:34 +10:00
Marc Durdin
b17805df3a feat(core): add 'name' section
Updates compiler, headers and documentation for the 'name' section,
which will always be present as the LDML keyboard spec requires at least
one name in the `<names>` element.
2022-09-04 05:45:43 +10:00
Marc Durdin
ab999ea0ba chore(common): cleanup basic.txt a bit more
Also uses `index()` to get count of strings automatically.

Reformat basic.txt to make it more readable as it will get less
'basic' over time!
2022-09-03 06:36:41 +10:00
Marc Durdin
738401cdb5 feat(common): hextobin index command
Adds an `index` command as suggested by #7186.

> Future enhancement suggestion: include ... index() for block indices
> within these repeated blocks. Would be very useful for string tables
> in the example ref.
2022-09-03 06:28:22 +10:00
Marc Durdin
aef38f12df chore(core): define zero length string vs null string for kmxplus 2022-09-03 06:11:24 +10:00
Marc Durdin
702db90a71 feat(core): add version field to meta
Also adds semver package for validating the version field.
2022-09-02 14:13:10 +10:00
Marc Durdin
17371c3801 feat(common): add size and offset calculators to hextobin
This makes it easier to write a binary file which contains offsets and
size values. Using this for unit tests for the LDML keyboard compiler.

Example kmldmlc basic.txt is updated and included.

Future enhancement suggestion: include count() for repeated blocks and
index() for block indices within these repeated blocks. Would be very
useful for string tables in the example ref.
2022-09-02 11:37:59 +10:00
Marc Durdin
918d261f99 feat(developer): add conformsTo validity unit test
This is reasonably random, but wanted to add another test for content
rather than structure, just to verify that the DTD-XSD-jsonschema
translation is doing the right thing according to what we expect.

Looking good.
2022-09-02 08:01:48 +10:00
Marc Durdin
25b2d255aa feat(developer): use json schema to verify source file
As we do not appear to have access to a good DTD validation library in
Node.js, we will instead convert the DTD to a JSON schema file and use
that for validation.

I have not verified every aspect of the JSON schema at this point, but
will incrementally do so as I implement further aspects of the compiler.

This updates all the testing to use the schema. As the schema validation
is slow, I have increased the 'normal' time for tests.
2022-09-01 16:56:50 +10:00
Marc Durdin
14094c8979 feat(developer): add kmlmc and kmldmlc to make test
Also removes `-test` from normal `make build` for kmlmc, moving that
responsibility to `make test` (but uses `-tdd` to avoid rebuild).
2022-09-01 15:28:41 +10:00
Marc Durdin
7d38e26b09
chore: Apply suggestions from code review 2022-08-31 14:46:27 +10:00
Marc Durdin
1c0f68bb75 feat(developer): loca minimization + cleanup kmldmlc error list
Yeah two for the price of one:

- minimize locales at compile time
- cleans up the list of errors to make it easier to maintain and use
2022-08-31 14:30:03 +10:00
Marc Durdin
17c221f4ce chore: remove comment 2022-08-31 10:21:03 +10:00
Marc Durdin
b58bb36d80 feat(developer): add loca tests
Note: bumps target to es2020 for better Intl.* support. Verified as
supported by Node 16+ and modern browsers.

Adds support for multiple locales and basic canonicalization. Does not
support normalization of locales.
2022-08-31 10:18:57 +10:00
Marc Durdin
5d44d2ecc7 chore(developer): Merge branch 'chore/merge-master-into-feature-ldml' into feat/developer/kmldmlc-meta-tests-and-cleanup 2022-08-31 10:10:03 +10:00
Marc Durdin
8fc7921ed4 chore(developer): cleanup todo comments 2022-08-30 13:57:16 +10:00
Marc Durdin
995d731ccb feat(developer): add unit tests for meta compiler
First, a little bit of refactoring to make it cleaner to do unit test
for each section.

Then add some global error values.

Throw in a helper function or two.

Dust off the edges of the CompilerCallbacks object.

Duct tape together validation of the normalization attribute.

And ... eventually we get a nice clean-like unit test for the meta
compiler, with three pretty little fixtures and as-neat-as-you-like-it
assertions.
2022-08-30 13:52:36 +10:00
Marc Durdin
db11553c9a refactor(developer): each section compiler is a pure function
Well, almost. There are still callback side-effects for messages. These
may in future be returned as an array as well, eliminating the need for
a message callback.

But restructures each section compiler so that it is self-contained and
responsible only for writing its own section. Also tightens up the types
for the sections constants.

Last, but not least, moves responsibility for writing the output .kmx
out of compiler.ts and into its callers -- removing one dependency from
the compiler. Reading input files still needs to be done in the
compiler, as there may be multiple input files with `<import>` that we
cannot know about until we start parsing xml.
2022-08-30 11:13:11 +10:00
Marc Durdin
c0147dc6ea refactor(developer): separate files for kmldmlc sections
@keymanapp-test-bot skip
2022-08-30 07:16:53 +10:00
Marc Durdin
8dc68edc6c
Merge pull request #7140 from keymanapp/feat/developer/unit-tests-for-kmldmlc
feat(developer): unit test framework for kmldmlc 🙀
2022-08-29 15:16:41 -05:00
Marc Durdin
c85b1da333
Merge pull request #7129 from keymanapp/feat/core/use-ldml-keyboard-compiler-for-tests
feat(core): use kmldmlc to build test keyboards 🙀
2022-08-29 15:15:55 -05:00
Marc Durdin
5dc5b9325c
Merge pull request #7000 from keymanapp/chore/developer/keymanweb-compiler-remove-unused-dependencies
chore(developer): remove unused dependencies from KeymanWeb compiler 🛩
2022-08-29 14:48:46 -05:00
Marc Durdin
a72146e42d feat(developer): unit test framework for kmldmlc 2022-08-29 17:21:05 +10:00
Marc Durdin
b31a8755c0
Merge pull request #7127 from keymanapp/feat/developer/ldml-compiler-xml-loader-and-compiler
feat(developer): LDML keyboard XML Loader and Compiler 🙀
2022-08-27 14:59:55 -05:00
Marc Durdin
496ea0641a
chore: add comment for BUILDER_SECT
Co-authored-by: Steven R. Loomis <srl295@gmail.com>
2022-08-27 09:03:29 +10:00
Marc Durdin
b50d03b7d6 chore(developer): chmod build.sh 2022-08-26 15:26:36 +10:00
Marc Durdin
daf6a7970e feat(core): use kmldmlc to build test keyboards
Removes .kmx from repo and builds them from the source .xml files using
kmldmlc (building that if necessary also with a poor fella's dependency
check for now), and generally a lot of cleanup of the meson.build files
for the ldml tests.

Moves keyboards to a subfolder of the ldml unit test folder to keep them
neat and tidy and avoid embedding huge files into ldml.js for test of
WASM.

Note: 000_null_keyboard and 002_null_invalid do build successfully, but
the core ldml keyboardprocessor currently fails on them, because they
are missing keys and vkey sections.
2022-08-26 15:12:38 +10:00
Marc Durdin
dc172cb981 chore(developer): robustness in ldml keyboard compiler 2022-08-26 09:06:12 +10:00
Marc Durdin
3c120ebae7 feat(developer): LDML keyboard XML Loader and Compiler
Implements basic LDML keyboard xml loader and compiler.
2022-08-26 08:51:15 +10:00
Marc Durdin
3865870114 chore(developer): add padding to loca and vkey 2022-08-26 05:23:11 +10:00
Marc Durdin
7644ce668f chore(developer): cleanup for kmx-builder 2022-08-26 04:44:00 +10:00
Marc Durdin
420dc1c10c feat(developer): emit kmxplus format 2022-08-26 04:40:25 +10:00
Marc Durdin
23d9aad0a0 chore(developer): cleanup 2022-08-25 15:12:00 +10:00
Marc Durdin
aed8f0d11a feat(developer): Write kmxplus section
Adds support for writing kmxplus section of a kmx file. Extends KMXFile
to a KMXPlusFile subclass. Adds KMXPlusBuilder which is called from
KMXBuilder if KMXBuilder finds that the file is a KMXPlusFile.

This means that KMXPlusBuilder generates a buffer that could be stored
independently of its parent .kmx file transport, if that is later
considered desirable.

Adds new constants for sizes of various table components, which makes
building the file much simpler.
2022-08-25 12:26:21 +10:00
Marc Durdin
6aeb20e77c
Revert "Revert "feat(core): turn @keymanapp/ldml-keyboard-constants into a module 🙀"" 2022-08-24 14:32:22 +10:00
Marc Durdin
8ebae9515e
Revert "feat(core): turn @keymanapp/ldml-keyboard-constants into a module 🙀" 2022-08-24 14:26:44 +10:00
Marc Durdin
5e296c98af feat(core): turn @keymanapp/ldml-keyboard-constants into a module
Splits the building of keyboardprocessor_ldml.h into a separate
tsconfig.build.json, and designates keyboardprocessor_ldml.ts as a new
Typescript/Node module @keymanapp/ldml-keyboard-constants.

Includes an example usage in the (currently unused) kmx-plus.ts.
2022-08-24 14:10:20 +10:00
Marc Durdin
a3e672e808 feat(developer): LDML compiler kmx-builder
Generates a valid .kmx file (legacy structures only) from an
intermediate in-memory structure. Included for completeness, although it
is anticipated that LDML keyboards will not have any groups, though they
will probably have some metadata stores.
2022-08-24 13:43:09 +10:00
Marc Durdin
d4553f674a chore(developer): rename structures 2022-08-24 09:42:08 +10:00
Marc Durdin
4018e528d9 feat(developer): groundwork - kmx file structures
Defines kmx binary structures and wireframe builder to emit a .kmx file.

Currently the builder will emit a roughly correct header. Next steps are
to build a .kmx file with stores, groups and keys, to match existing KMX
compiler behaviour. Once that is done, will return to generation of
KMXPlus data structures.

Compiler is designed to work independent of file system so it should run
inside browser or node environment. kmldmlc.ts is node-specific, but all
other modules should be multi-platform.
2022-08-24 09:32:30 +10:00
Marc Durdin
9ce6892112
Merge pull request #7101 from keymanapp/feat/developer/ldml-keyboard-ts-structures
feat(developer): LDML keyboard compiler - infrastructure 🙀
2022-08-23 18:29:49 -05:00
Marc Durdin
8cba43f989
chore: Apply suggestions from code review
Co-authored-by: Darcy Wong <darcy_wong@sil.org>
Co-authored-by: Steven R. Loomis <srl295@gmail.com>
2022-08-24 09:29:03 +10:00
Marc Durdin
e6ff47f2b3 chore: Merge branch 'master' into feature-ldml 2022-08-24 04:21:21 +10:00
Marc Durdin
cc8875dc9e chore(developer): cleanup comments 2022-08-24 03:54:25 +10:00
Marc Durdin
c108e5ecd6 feat(developer): add kmldmlc to developer installer 2022-08-24 03:29:45 +10:00