Commit graph

43 commits

Author SHA1 Message Date
Joshua A. Horton
fef5b3ab63 chore(web): Merge branch 'feat/web/wordbreaker-property-data-gen' into feat/web/wordbreaker-data-optimization 2024-08-08 12:56:46 +07:00
Joshua A. Horton
7016e5b40b chore(common/models): removes incomplete early check-in 2024-08-08 12:53:04 +07:00
Joshua A. Horton
dda43aa399 change(common/models): moves wordbreaker data-gen from src/ to tools/, removes old data.ts 2024-08-08 12:51:30 +07:00
Joshua A. Horton
b145adfaf8 fix(web): fixes value for end of BMP range 2024-08-07 09:52:10 +07:00
Joshua A. Horton
fdecc602d3 feat(common/models): connect encoded lookup table 2024-08-07 09:37:22 +07:00
Joshua A. Horton
5045df13db change(common/models): improve encoding format 2024-08-07 09:37:07 +07:00
Joshua A. Horton
23c03082a5 chore(web): Merge branch 'feat/web/wordbreaker-property-data-gen' into feat/web/wordbreaker-data-optimization 2024-08-07 08:52:21 +07:00
Joshua A. Horton
217ad29a5b fix(common/models): actually run the data-compiler 2024-08-07 08:36:25 +07:00
Joshua A. Horton
e2dffff68d fix(common): fix data-compiler path references 2024-08-07 08:36:17 +07:00
Joshua A. Horton
c7f4631fea fix(common/models): fix tsconfig issue from this PR's changes 2024-08-07 08:34:57 +07:00
Joshua A. Horton
77b94bf559 feat(common/models): integrates wordbreaking-data processor into wordbreaker toolchain 2024-08-06 14:47:04 +07:00
Joshua A. Horton
cc023dd5af chore(common/models): reverts unneeded addition of zlib 2024-08-06 13:36:30 +07:00
Joshua A. Horton
cf4f832817 chore(web): Merge branch 'master' into feat/web/wordbreaker-property-data-gen 2024-08-06 10:01:41 +07:00
Joshua A. Horton
d66d474f64 fix(web): add limited Array.from polyfill for lm-worker use
Fixes: #11502
Fixes: KEYMAN-WEB-K4
2024-06-07 11:56:18 +07:00
Joshua A. Horton
0b79b86197 feat(common/models): initial pass for encoded-string wordbreaker data tables 2024-02-13 08:29:38 +07:00
Joshua A. Horton
58c7575d4b feat(web): full property data-table generation functionality; data is exact match 2024-02-13 08:29:25 +07:00
Joshua A. Horton
901861a75d feat(web): implements string-encoded prop-array search 2024-02-13 08:29:25 +07:00
Joshua A. Horton
9093c0719b change(web): preps for wordbreaker child projects 2024-02-13 08:29:24 +07:00
Joshua A. Horton
fed4618d02 feat(common/models): tslib use, treeshaking in lm-worker 2023-06-08 14:39:55 +07:00
Joshua A. Horton
dce763600e chore(common/models): lm-worker import path cleanup 2023-02-03 14:52:12 +07:00
Joshua A. Horton
b773cc852d change(common/models): wordbreakers -> ES modules 2023-01-20 13:26:09 +07:00
Joshua A. Horton
26fb6da3ab feat(common/models): adds tokenization tests for wordbreaks near the caret 2022-09-15 13:23:21 +07:00
Joshua A. Horton
94face23d0 chore(common/models): minor cleanup 2022-09-13 11:06:54 +07:00
Joshua A. Horton
261e252026 docs(common/models): adds minor doc re 'sot', 'eot' 2022-09-13 10:38:34 +07:00
Joshua A. Horton
8e076fe3fd feat(common/models): actual wordbreaker customization 2022-09-13 10:18:24 +07:00
Joshua A. Horton
51caf533a8 feat(common/models): interface for custom breaker rules 2022-09-12 15:14:28 +07:00
Joshua A. Horton
991a5bc271 feat(common/models): wordbreaker context .match method 2022-09-12 15:14:27 +07:00
Joshua A. Horton
5bad267e11 refactor(common/models): default wordbreaker state tracking 2022-09-12 15:14:26 +07:00
Joshua A. Horton
91448c344f feat(web): more default-breaker unit tests 2022-09-06 14:52:16 +07:00
Joshua A. Horton
a6e65333c5 fix(common/models): wordbreaker test require paths 2022-09-06 08:54:56 +07:00
Marc Durdin
e0b308b983 chore(web): cleanup for ts projects 2022-04-13 15:19:55 +10:00
Marc Durdin
4d94786852 chore(web): replace lerna with npm workspaces and ts projects
Fixes #6320.

Part 1 of moving from lerna to a simpler, maintained monorepo solution,
using TypeScript Projects and NPM Workspaces.

There is more work to be done here. At this point, KeymanWeb builds and
runs without errors, but the built file is substantially different,
mostly in include order.

Using TypeScript Projects, we move away from the need to run build
scripts in various locations for almost all the Typescript modules.

TODO: Embedded versions and tests have not been verified.

TODO: developer/server is not yet verified.

TODO: developer/js (needs a rename!) is not yet verified.

TODO: Currently, the predictive-text folder needs refactoring to move
the construction of the worker wrapper out of the Predictive Text build
and into the final assembly of keymanweb.js (as it should be valid to
run it as a separate .js anyway).

TODO: Most of the `<reference>` paths need to be re-verified. Ideally
there should be no references outside the current module for any given
.ts.

TODO: The embedded vs browser vs node (headless) builds should be tidied
up for consistency so that it's obvious what depends on what. This is
currently messiest in the predictive-text folder, where the output names
diverge from the filenames and the various files are mixed in the same
folder (as evidenced by the exclusions listed in each tsconfig.json).

TODO: `npm install` should be removed from most build scripts and
instead `npm ci` (#6196) should be run only once from the top-level
folder for any given build. I've had eliminated side-effects from the
`install` action for npm, which makes it easier to reason about state.

TODO: verify_npm_setup and related functions can probably be eliminated.

TODO: most of the build scripts should be largely eliminated for web.

TODO: several ts projects use inconsistent output folders.

TODO: it may be possible to generate a .d.ts for models/types so that
we can use a consistent reference for those as well.

TODO: build.sh, tsconfig.json should always be in the module's top-level
folder, not in a subfolder such as src (e.g. see input-processor/src,
keyboard-processor/src, web/source).

TODO: resources/web-environment should be in common/web.

TODO: other js node_modules imports should be wrapped like es6-shim.

TODO: fix up the publish code for npm modules

TODO: eliminate version numbers from package.json if possible?

Whew, that's most of the stuff I noticed!
2022-04-12 21:02:33 +10:00
Eddie Antonio Santos
d2249ad2d7
change(common/models/wordbreakers): remove extendedPictographic regexp (unused) 2020-07-17 15:35:37 -06:00
Eddie Antonio Santos
37e9b034e4
fix(common/models/wordbreakers): use ES5 compatible regexp 2020-07-17 15:23:18 -06:00
Eddie Antonio Santos
dbb2908aa0
change(common/models/wordbreakers): update word boundary props to 13.0
This updates the Word_Boundary table to Unicode 13.0.0.

The `searchForProperty()` function has also been changed so that the END
property of a range is no longer needed; all code points are explicitly
covered in the table. Consequently, ranges containing unmapped code points
(WordBreakProperty.Other) are explicit.
2020-07-17 15:03:44 -06:00
Eddie Antonio Santos
a4dae62817
refactor(common/lmlayer): move default word breaker to models-wordbreakers package 2020-05-19 15:07:06 -06:00
Eddie Antonio Santos
0baf04ae06
refactor(common/lmlayer): move placeholder wordbreaker to models-wordbreakers 2020-05-19 15:04:29 -06:00
Eddie Antonio Santos
e98ae7d7a1
fix(common/lmlayer): add @types/node to @keymanapp/models-wordbreaker to prevent failure 2020-05-19 14:57:55 -06:00
Eddie Antonio Santos
7b1f86b399
fix(common/lmlayer): export wordBreakers namespace 2020-05-19 14:46:33 -06:00
Eddie Antonio Santos
863b83883b
chore(common/lmlayer): get ASCII word breaker compiling 2020-05-19 14:36:03 -06:00
Eddie Antonio Santos
374f0dfd8e
Revert "refactor(common/lmlayer): move placeholder word breaker to @keymanapp/lexical-model-word-breakers"
This reverts commit 73b16ae33e.
2020-05-19 14:34:11 -06:00
Eddie Antonio Santos
6fc39df5d0
Revert "refactor(common/lmlayer): move default word-breaker to @keymanapp/lexical-model-word-breakers"
This reverts commit 1f4de1ba93.
2020-05-19 14:32:19 -06:00
Eddie Antonio Santos
73b5d520e2
change(common/lmlayer): move to common/models/wordbreakers 2020-05-19 14:08:56 -06:00