Commit graph

292 commits

Author SHA1 Message Date
Marc Durdin
f3fb3ac87b maint(common): move build-utils-ci.inc.sh to ci/ci-publish.inc.sh
Removes a number of unused functions from the file, renames remaining
functions to match our builder script prefix patterns, removes unused
imports from various build scripts (e.g. Android).

Fixes: #14453
2025-08-05 12:54:56 +10:00
Marc Durdin
acefdf281a maint(common): move typescript script functions into typescript.inc.sh
Fixes: #14448
Build-bot: build
Test-bot: skip
2025-08-03 07:01:57 +10:00
Marc Durdin
2726050601 maint(common): rename and move node-related script functions into node.inc.sh
Consolidates the node-related script functions into node.inc.sh, as part
of cleaning up the build scripts and making them easier to maintain into
the future.

Fixes: #14447
2025-08-02 13:34:29 +10:00
Marc Durdin
f394245636 maint(common): consolidate builder scripts
Clarifies the confusing builder.inc.sh / build-utils.sh distinction by
giving the scripts more appropriate names. Most build scripts should use
builder-full.inc.sh; some helper scripts can use builder-basic.inc.sh.
Documented in resources/build/README.md.

Renames:
* resources/build/builder.inc.sh to resources/build/builder-full.inc.sh
* resources/build/build-utils.sh to resources/build/builder-basic.inc.sh

Other changes:
* Moves Android-specific functions out of builder-basic.inc.sh and into
  android/build.sh.
* Renames functions in builder-basic.inc.sh

More functions may be moved from builder-basic.inc.sh into utils.inc.sh
or other scripts in the future.

Fixes: #14065
Build-bot: build all
Test-bot: skip
2025-08-02 08:11:24 +10:00
Marc Durdin
65158a0526 maint(common): rename shellHelperFunctions.sh to utils.inc.sh
Adds documentation for various functions, removes unused functions (a
couple of simple, very lightly used functions were unDRYed; these could
go back the other way across all shell scripts if necessary). TODO items
noted for follow-up refactoring.

Fixes: #14275
Relates-to: #14269
Build-bot: build all
Test-bot: skip
2025-08-02 06:58:40 +10:00
Steven R. Loomis
0a35181bed chore(developer): rename x: to compileContext:
Fixes: #14117
2025-06-26 15:58:43 -05:00
Steven R. Loomis
28ba904b37 chore(developer): rename ObjectWithMetadata to ObjectWithCompileContext
Fixes: #14117
2025-06-26 15:56:52 -05:00
Steven R. Loomis
ba8dd1a17e chore(developer): document findInstanceObject() return value
per code review
Fixes: #14099
2025-06-21 14:44:07 -05:00
Steven R. Loomis
7a741dc9d0 chore(developer): ldml: additional line numbers
- ajv validation
- more fixes from out of box testing

Fixes: #14099
2025-06-02 16:30:29 -05:00
Steven R. Loomis
fab29aad77 feat(developer): ldml: update errors for unparseable reorder
Fixes: #14091
2025-06-02 08:52:00 -05:00
Steven R. Loomis
00b4de5f53 feat(developer): add line numbers to tran compiler
- move ObjectWithMetadata into symbol utils
- don't validate transforms twice (tran + bksp)

This is for #13937 but does not resolve all items
2025-05-22 17:03:40 -05:00
Joshua Horton
155c1a2b12 maint: update mocha
Some of the npm audit warnings left after #13897 require an updated `mocha` version to remedy.  As we only really use `mocha` to do unit-testing, it should be safe to update if all of our existing tests pass with the new version.

Test-bot: skip
2025-05-13 14:02:02 +07:00
Steven R. Loomis
b23275d7e1 feat(developer): fix imports in ldml-keyboard-xml-reader.ts
Fixes: #10622
2025-05-09 13:32:15 -05:00
Steven R. Loomis
8ec8fb1890 feat(developer): update for line numbers
- annotate why we're calling SymbolUtils.removeSymbols()

Fixes: #10622
2025-05-09 13:09:53 -05:00
Steven R. Loomis
6194392310 feat(developer): update for line numbers
- reduce whitespace churn
- other review comments

Fixes: #10622

Co-authored-by: Marc Durdin <marc@durdin.net>
2025-05-09 13:07:51 -05:00
Steven R. Loomis
8e2eb498e3 feat(developer): update for line numbers
- reduce whitespace churn

Fixes: #10622
2025-05-09 11:41:23 -05:00
Steven R. Loomis
307288dc3b chore(developer): deps: fast-xml-parser@5.2.2
My line number changes were merged into the main branch

Fixes: #10622
2025-05-08 17:43:19 -05:00
Steven R. Loomis
ad0ea8678b feat(developer): update for line numbers
- added new DelegatingCompilerCallbacks class
- added ResolvingCompilerCallbacks which does event resolution
- added the XML file name metadata to *every object* after XML parse.

Fixes: #10622
2025-05-08 17:29:30 -05:00
Steven R. Loomis
a401d7eec3 feat(developer): update for line numbers
- make line finder cache lazy loading
- rename LDMLResolver to LineFinderResolver

Fixes: #10622
2025-05-05 09:17:36 -05:00
Steven R. Loomis
0edbbec981 Merge remote-tracking branch 'upstream/master' into feat/developer/10622-xml-parse-line-numbers-epic-ldml 2025-04-26 14:41:54 -05:00
Marc Durdin
4f829f5c8f chore: Merge remote-tracking branch 'origin/master' into chore/merge-beta-to-master-b18s5 2025-04-21 07:13:35 +07:00
Steven R. Loomis
e003612ea0 feat(developer): update for line numbers
- remove resolver complexity in callback subclasses
- call LDMLKeyboardXMLSourceFileReader.eventResolver.resolve() statically from ldml-compiler-messages

Fixes: #10622
2025-04-10 17:22:51 -05:00
Steven R. Loomis
64bcfc66e9 Merge remote-tracking branch 'upstream/master' into feat/developer/10622-xml-parse-line-numbers-epic-ldml 2025-04-10 17:02:05 -05:00
Steven R. Loomis
53fd228d08 feat(developer): update for line numbers
- fix windows path specific issue in test

Fixes: #10622
2025-04-03 17:24:57 -05:00
Steven R. Loomis
11f77ed876 feat(developer): update for line numbers
- fix ts issue on reader

Fixes: #10622
2025-04-03 16:47:04 -05:00
Steven R. Loomis
8d438283c6 feat(developer): update for line numbers
- split out LineFinder to separate class

Fixes: #10622
2025-04-03 16:40:53 -05:00
Steven R. Loomis
1f2b8463f5 chore(developer): deps: update to fast-xml-parser@5.1.0 (Metadata branch)
Fixes: #10622
2025-04-03 12:15:20 -05:00
Marc Durdin
21486882eb
Apply suggestions from code review 2025-04-03 16:44:36 +02:00
Marc Durdin
838aea52ed refactor(developer): consolidate project loader
Consolidates project loading code from kmc and kmc-copy, in preparation
for kmc-test epic. This leaves open a change we need to make to
CompilerCallbacks, to consolidate the fsAsync and net callbacks into
CompilerCallbacks as a first-class part of CompilerCallbacks. We will
then need to start to move all the fs and net callback interactions to
the async model across all kmc modules, at some point.
2025-04-03 11:37:54 +07:00
Marc Durdin
cf16c37869 refactor(developer): move sentry and developer options into console apps
Move the sentry dependency out of developer-utils and into the console
apps kmc and server, in process of making sure that developer-utils has
no node dependencies.

Also copy the options.ts module into kmc and server, for the same
reason.

This first step duplicates two units, options.ts and KeymanSentry.ts. We
do need to refactor this again later to avoid this duplication (maybe
via yet another internal package?)
2025-04-03 09:56:46 +07:00
Marc Durdin
70838ff1ac refactor(developer): rename to developer-utils-messages.ts
Renames common-messages.ts to developer-utils-messages.ts. This was
split from /common/web/types, but the messages module retained the old
name, which was confusing (two modules with same name).
2025-04-03 09:26:27 +07:00
Marc Durdin
ac6eb0b767 chore: use explicit node: prefix for node imports
This makes it easier for us to enumerate modules that have node
dependencies. There may be others I have not yet picked up.
2025-04-03 08:01:08 +07:00
Marc Durdin
0f05da957f fix(developer): remove redundant Name and RTL fields from .kps LexicalModel
Follows on from the similar changes to `Keyboard` fields in #13600. This
deprecates the `RTL` and `Name` fields. `RTL` was never used. `Name` was
written but never presented to end users, so effectively irrelevant.
`Name` also was present in `File.Description` (already deprecated) for
the corresponding model.js _and_ `Info.Name`, so doubly redundant.

Also updates the schema documentation to mark the deprecated fields.

Relates-to: #13600
2025-04-02 05:39:09 +07:00
Steven R. Loomis
29f387bebe Merge remote-tracking branch 'upstream/master' into feat/developer/10622-xml-parse-line-numbers-epic-ldml 2025-03-31 16:16:10 -05:00
Steven R. Loomis
0c290bd37a feat(developer): update for line numbers
- update to metadata branch of fast-xml-parser
- restructure metadata handling
- use type safety for fast-xml-parser options

Fixes: #10622
2025-03-31 16:11:19 -05:00
Marc Durdin
a52437c407 chore: Merge branch 'master' into chore/b18s4-merge-beta-to-master 2025-03-29 05:38:08 +07:00
Marc Durdin
52d016fe7c chore(developer): remove redundant fields from kps reader unit test
Note: the round-trip test still passes because the XML data passes
basically unmodified through, even though we make no reference to the
fields in the code.

Fixes: #13576
2025-03-28 14:27:08 +07:00
Marc Durdin
9dabcc0306 fix(developer): remove redundant keyboard fields from .kps
The fields 'name', 'version', and 'rtl' were defined in the .kps file,
but were not actually used by the compiler, as kmc-package reads the
metadata from the keyboard file rather than relying on this data, which
often is out of date. This led to confusion for keyboard authors, so
this change removes that metadata from the package.

The one visible side-effect (apart from the data disappearing from
future .kps files) is that the package editor no longer shows this
metadata in the Keyboards view (but as it may be out-of-date, this is
probably a good thing).

Fixes: #13576
Fixes: #13593
2025-03-28 14:07:23 +07:00
Steven R. Loomis
8aa9219410 Merge remote-tracking branch 'upstream/master' into feat/developer/10622-xml-parse-line-numbers-epic-ldml 2025-03-27 16:02:45 -05:00
Marc Durdin
bc4462a312
Merge pull request #13532 from keymanapp/fix/developer/13455-ERROR_TextBeforeOrAfterNulInOutput
fix(developer): add validation of `nul` usage in output part of rule
2025-03-19 06:12:54 +07:00
Marc Durdin
5683b370fe fix(developer): add validation of nul usage in output part of rule
If `nul` is used in the output part of a rule, then text or
text-emitting statements cannot also be included in the output of that
rule. This was not enforced in earlier versions of Keyman.

Adds a unit test for the enhanced validation.

Fixes: #13455
Relates-to: keymanapp/keyboards#3379
2025-03-18 09:54:06 +07:00
Marc Durdin
d4540e3c68 chore(developer): use caps for constants in xml-utils 2025-03-17 12:19:10 +07:00
Marc Durdin
7d5c98ddba fix(developer): bump fast-xml-parser to 5.0.9
This is a major dependency version bump (4.5.0 to 5.0.9) which we would
normally avoid during beta. However, we need the fix
NaturalIntelligence/fast-xml-parser#725 for hex escapes in XML, which is
incorporated in 5.0.9.

I have assessed the other changes to fast-xml-parser and found no
breaking changes for us, but I tightened the types in xml-utils
declarations and found some minor inconsistencies which appear to have
no impact, and which I have corrected:
* wrong type in unused parameter to `tagValueProcessor`
* reference to unused property `options.emptyTag`

While doing this, I consolidated the common options for the parser in
order to verify consistency, but made no changes to the resolved
parsing/building options.

Added a test to ldml keyboard reading, to verify that numeric strings
are treated as strings, given the divergence in the `numberParseOptions`
option, and it shows that numeric strings are treated as strings.

Fixes: #13348
2025-03-17 06:22:29 +07:00
Marc Durdin
96a96bef15 fix(developer): add units tests to verify non-BMP numeric entities in XML reader
fast-xml-reader has a bug with numeric entities. See:
  https://github.com/NaturalIntelligence/fast-xml-parser/issues/725

This commit adds a unit test to verify that non-BMP numeric entities
will be parsed correctly. It will fail until we update the
fast-xml-parser dependency.

Relates-to: #13348
2025-03-17 05:50:47 +07:00
Marc Durdin
1a0b830c0b
Merge branch 'master' into chore/b18s3-merge-beta-to-master 2025-03-14 08:54:50 +07:00
Marc Durdin
d2b457b2e4 fix(developer): load Keyman Developer options in Server before use
Fixes: #13420
2025-03-10 06:28:39 +07:00
Marc Durdin
060189a1ab chore(developer): drop coverage threshold for developer-utils to 45% 2025-03-05 00:24:41 +03:00
Steven R. Loomis
0ba005a55b feat(developer): line number symbols
- improve and simplify lookup algorithm
- add test for override filename on imports

Fixes: #10622
2025-03-04 09:34:47 -06:00
Steven R. Loomis
2dca879aae
Merge branch 'master' into feat/developer/10622-xml-parse-line-numbers-epic-ldml 2025-03-03 17:15:48 -08:00
Steven R. Loomis
92e16b90a0 feat(developer): line number symbols
- move file read bottleneck out of compiler.ts into the LDML XML reader
- move the eventresolver into the LDML XML reader
- add XML_FILENAME_SYMBOL - for tagging when an object is from a different file than the default
(such as with an import)

Fixes: #10622
2025-03-03 19:12:34 -06:00