Commit graph

2740 commits

Author SHA1 Message Date
Marc Durdin
5ca59bfd1e chore(developer): cleanup API interfaces and types for kmcmplib
Introduces kmcmplibapi.h which declares all external functions and types
for kmcmplib, and updates kmcmplib and kmcmpdll to use these
declarations consistently.

Note that kmcmp_CompileKeyboardToBuffer declares a void* instead of
PFILE_KEYBOARD, because this is a large and complex type defined in both
kmcmpdll and kmcmplib. In the future, when kmcmpdll existing legacy
compiler is removed, we can make the kmcmplib type more public.
2023-03-03 13:21:17 +07:00
Marc Durdin
d3782228d2 chore(developer): kmcmplib - rename kmcmpdll.h to reduce confusion 2023-03-03 10:20:16 +07:00
Marc Durdin
f3310ab081 chore(developer): remove unused version library reference 2023-03-03 09:17:41 +07:00
Marc Durdin
882aedc575 refactor(developer): kmcmplib json-validation.cpp remove windowsisms 2023-03-03 09:13:07 +07:00
Sabine
750eb6b96c chore(developer): CheckFilenameConsistency/CHINT_FilenameHasDifferingCase, rename new functions 2023-03-02 13:55:25 +01:00
Marc Durdin
8c22e6269e chore(developer): cleanup kmcompx build scripts per recent builder changes 2023-03-02 14:10:48 +07:00
Marc Durdin
eac90588dc
Merge pull request #8336 from keymanapp/feat/developer/kmcmplib-feature-flag-tike
feat(developer): add 'Use Legacy Compiler' option to TIKE 🚑
2023-03-02 17:59:55 +11:00
Marc Durdin
dbd296f761
Merge pull request #8335 from keymanapp/feat/developer/kmcmplib-feature-flag-kmcomp
feat(developer): add `-use-legacy-compiler` flag to kmcomp 🚑
2023-03-02 17:59:47 +11:00
Marc Durdin
1790ad36a3
Merge branch 'feature-kmcompx' into feat/developer/kmcmplib-feature-flag-kmcomp 2023-03-02 16:40:39 +11:00
Marc Durdin
0c07a6d7a3 chore(developer): Merge branch 'feature-kmcompx' into chore/developer/master-to-kmcompx-A17S7 2023-03-02 12:37:37 +07:00
Sabine
febafc6286 chore(developer): CheckFilenameConsistency/CHINT_FilenameHasDifferingCase, rename new functions 2023-03-01 17:01:47 +01:00
SabineSIL
048448a2a4
feat(developer): cross-platform kmx compiler 🚑 (#6980)
* get keymans version



Merge branch 'master' of https://github.com/keymanapp/keyman into refactor/developer/6026-kmcompx-compiler-cross-platform-part-2-NEW_VERSION

# Conflicts:
#	common/core/desktop/src/kmx/kmx_context.h
#	common/core/desktop/src/kmx/kmx_processor.cpp
#	common/windows/cpp/include/keymanversion.h
#	common/windows/cpp/include/kkmnkbd/Compfile.h
#	common/windows/cpp/include/kmtip_guids.h
#	common/windows/cpp/src/kmtip_guids.cpp
#	core/src/kmx/kmx_conversion.cpp
#	core/src/kmx/kmx_conversion.h
#	core/src/kmx/kmx_file.cpp
#	core/src/kmx/kmx_processevent.cpp
#	core/src/kmx/kmx_processor.hpp
#	core/src/kmx/kmx_xstring.cpp
#	developer/js/.gitignore
#	developer/js/package-lock.json
#	developer/js/tests/tsconfig.json
#	developer/src/kmcmpdll/CasedKeys.cpp
#	developer/src/kmcmpdll/CheckFilenameConsistency.cpp
#	developer/src/kmcmpdll/CheckFilenameConsistency.h
#	developer/src/kmcmpdll/CheckForDuplicates.cpp
#	developer/src/kmcmpdll/CheckForDuplicates.h
#	developer/src/kmcmpdll/CheckNCapsConsistency.cpp
#	developer/src/kmcmpdll/CheckNCapsConsistency.h
#	developer/src/kmcmpdll/Compiler.cpp
#	developer/src/kmcmpdll/DeprecationChecks.cpp
#	developer/src/kmcmpdll/NamedCodeConstants.cpp
#	developer/src/kmcmpdll/UnreachableRules.cpp
#	developer/src/kmcmpdll/UnreachableRules.h
#	developer/src/kmcmpdll/compfile.h
#	developer/src/kmcmpdll/kcframe/kcframe.cpp
#	developer/src/kmcmpdll/kcframe/kcframe.vcxproj
#	developer/src/kmcmpdll/kcframe/kcframe.vcxproj.filters
#	developer/src/kmcmpdll/kmcmpdll.vcxproj
#	developer/src/kmcmpdll/kmcmpdll.vcxproj.filters
#	developer/src/kmcmpdll/version.rc
#	developer/src/kmcmpdll/versioning.cpp
#	developer/src/kmlmc/bundle.sh
#	developer/src/kmlmc/tests/test-join-word-breaker.ts
#	developer/src/kmlmc/tests/test-override-script-defaults.ts
#	developer/src/server/tsconfig.json
#	windows/src/developer/kmcmpdll/Makefile
#	windows/src/developer/kmcmpdll/vkeys.h
#	windows/src/engine/keyman32/keymanengine.h
#	windows/src/global/inc/Vkeys.h
#	windows/src/global/inc/rc4.h
#	windows/src/global/vc/rc4.cpp

* changes in meson + include-paths of several files

* chore: remove common/core

* chore: remove unnecessary files

* chore: move headers and refs

* refactor(developer): fixup path to tests

* refactor(developer): fixup path to tests
tidied up +removed unnecessary files
checked contents of files+included a few new functions needed for myVersion
changed datatypes in CheckForDuplicates
tidy up more paths
CheckFileConsistency: not able to fully adapt for use of char16_t on non windows p.
	-> used char16_t + commented out some AddWarning().

* sorted out some #includes
renamed some functions to reduce ambiguity

* fix(developer): solved linker errors

* chore(developer): remove some comments

* chore(developer): introduced KMX_UCHAR

* chore(developer): restored whitespace, comments

* chore(developer): tidied up vkeys.H / .cpp

* chore(developer): remove keyboardprocessor...

* chore(common): refactor shared types into km_types.h

* chore(core): remove duplicate types

* chore(core): move kmx_file.h to common

* chore(developper): removed typedefs of kmcompcx.h

* chore(developer): removed kmcompx/include/kmx_file.h

* chore(developer): chanched include of pch.h

* chore(developer): removed files from kmcompx/include/

* chore(developer): removed commented sections ( ** old **) if files

* chore(developer): removed doubled/overwritten functions

* chore(developer): removed doubled/overwritten functions

* chore(developer): changed #includes ".." -> <..>

* chore(developer): changed #includes ".." -> <..>

* chore(developer): enable tests for keyboards repo for kmcompx

* chore(developer): new functions e.g. string_from_wstring...

* chore(developer): changed printf->u16sprintf

* chore(developer): fopen

* chore(developer): fixed errors on running kmcompxtest

* chore(developer): changef u16fmt

* chore(developer): fixup u16ncat

* chore(developer): added return Msgs for different types of errors while compiling

* chore(developer): tests using RTL run Ok ( changed RTu->RTL)

* chore(developer): replaced u16ncmp with u16nicmp at most places

* chore(developer): changed Version number for tests 16.046 to 15.0265

* chore(developer): mostly whitespace changes

* chore(developer): tidied up code (not finished yet)

* chore(developer): removed #pragma once from wrong places, edited #includes

* chore(developer): removed #pragma once from wrong places, edited #includes

* chore(developer): CheckFileNameConsistency changed to take KMX_WCHART const* (= wchar_t* const); used in compiler.cpp now

* chore(developer): excluded some tests from kmcompxtest (use get_test_source_exclude_some_kmn.bat now)

* chore(developer): Win-functions in FileNameConsistency

* chore(developer): Win-functions in FileNameConsistency

* chore(developer): CheckFNConsuistency: called u16printf with f´different par.

* chore(developer): tidied up code

* chore(developer): tidied up code

* chore(developer): tiedied up code (compiler.cpp)

* chore(developer): tiedied up code

* chore(developer): tidied up code

* chore(developer): removed all commentaries _S2

* chore(developer): tidied up code

* chore(developer): tidied up code

* chore(developer): tidied up code

* chore(developer): removed several Windows-specific functions

* chore(developer): removed unused datatype typedefs

* chore(developer): new GetCompilerErrorString() used in AddCompileMessage()

* chore(developer): new GetCompilerErrorString() used in AddCompileMessage()

* chore(developer): removed CWARN/CHINT out of CompMsg

* chore(developer): adapted kmcompxtest to test kmn-files that have errors included

* chore(developer): changed gitignore

* chore(developer): changed gitignore

* chore(developer): changed Version Number fot Tests 15.0.265.0 -> 15.0.270.0

* chore(developer): added tests for kmcompx

* chore(developer): move tests (for ErrorMessages) to common/test/keyboards

* chore(developer): move CERR-tests into test/keyboards/invalid

* chore(developer): change mostly whitespace changes

* chore(developer): change mostly whitespace changes

* chore(developer): delete files of keyboards/kmcompx_tests

* chore(developer): include check for correct Filename + rename var + change whitespace changes

* chore(developer): revert changes in AddCompileMessage, tidy up CreateTempFile

* chore(developer): remove python from meson.build

* chore(developer): change ErrExtra, change CompileMsg, change _finddata in FileExists

* chore(developer): check for '_' after Errornumber in Filename

* chore(developer): re-include AddCompilerVersionStore in Compiler.cpp

* chore(developer): add Message CERR_InvalidCharacter to UTF16TempFromUTF8() ; change #include <> -> ""

* chore(developer): add some more keyboardtests

* chore(developer): add some more keyboardtests

* chore(developer): change remaining old datatype-names to new names ( e.g. char -> KMX_CHAR )

* chore(developer): edit + add CERR_tests

* chore(developer): edit + add CERR_tests

* chore(developer): use char16_t in CheckFileNameConsistency

* chore(developer): comments: converting functions in kmx_u16

* chore: reorganize

* chore: commit kmcompx-files

* chore: use ifdef for vkeys

* chore: copy files back

* chore: more cleanup

* chore: meson build for kmcmplib

* chore: tweak unit test, error messages, and paths

* chore(developer): make kmcmplib a static library

* chore(developer): start kmcmpdll link to kmcmplib

* chore(developer): add build.sh for kmcmplib

* chore(developer): link kmcmpdll to kmcmplib

* chore(common): avoid building 'invalid' shared keyboards

* chore(developer): include BEGIN_LINE (repeated Begin)

* chore(developer): add more CERR_tests and move some to other folder, edit readme

* chore(developer): addCompilerMsg for RepeatedBegin

* chore(developer):repeatedBEgin plus rename functions to prevent same names in kmcmpdll and kmcmplib

* chore(developer): add flag flag_use_kmcompx as switch to use old code (kmcmpdll) or new code kmcmplib

* chore(developer): introduce namespace kmcmp instead of different filenames

* chore(developer): more changes for using namespaces

* chore(developer): solve merge conflicts in namespaces

* chore(developer): use namespaces in kmcmplib (except for in dllexport-functions)

* chore(developer): remove some unnecessary CERR_ tests

* chore(developer): remove CERR_CallIsProfessionalFeature, CERR_IncludeCodesIsProfessionalFeature, CERR_MnemonicLayoutIsProfessionalFeature

* chore(developer): note about func.call_js; change all "VESION x.y " to "VESION x.y or higher" ; rename flag for kmcompx to flag_use_new_kmcomp

* chore(developer): use PKMX_DWORD/PKMX_WCHAR   instead of LPKMX_DWORD/LPKMX_WCHAR; use PKMX_STR instead of char* in CompilerMessageProc

* chore(developer): remove kmx-file; NamedCodeConstants: rename variables, set const variables for buf sizes, brackets in for-loops

* chore(developer): change include paths ../

* chore(developer): get rid of reopened namespaces of kmcmp

* chore(developer): restored kmcompxtest

* chore(developer): change #include paths ../ ; excange km_kbp_cp with KMX_WCHAR; change some variable names

* chore(developer): use environment var KEYMAN_ROOT in path for tests; remove try-catch from json-validation.cpp

* chore(developer): use environment var KEYMAN_ROOT in path for tests; remove try-catch from json-validation.cpp

* chore(developer): revert change in NamedCodeConstants

* chore(developer): restored kmcompxtest, adapted readme

* chore(developer): use res-fils/version.rc for kmcmplib

* chore(developer): rename some functions/variables

* chore(developer): typos, comments, add function u16rchr()

* chore(developer): typos, comments, add function u16rchr()

* chore(developer): rename Functions, version rc-files

* chore(developer): rename Functions, version rc-files, set flag_use_new_kmcomp to use (old) kmcmpdll

* chore(developer): add BeginLine in kmcmpdll/Compiler.cpp; fread() in kmcmp_CompileKeyboardFileToBuffer

* chore(developer): set flag_use_new_kmcomp to only use kmcmpdll

* chore(developer): set flag_use_new_kmcomp to only use kmcmpdll...and back to kmcmplib

* (chore(developer): set flag_use_new_kmcomp to only use kmcmpdll...and back

* (chore(developer): changes in NamedCodeConstants::reindex() to run hieroglyphic.kmn properly

* chore(developer): changes in UnrechableRules::VerifyUnreachableRules() to run test_6440_unreachable_code..kmn

* chore(developer): remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta

* chore(developer): remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta, CWARN_ OK

* chore(developer): after stash:remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta, CWARN_ OK

* chore(developer): replace some #include <> with "" ; rename ErrExtra -> ErrExtraW

* chore(developer): replace some #include <> with "" ; rename ErrExtra -> ErrExtraW

* chore(developer): new converting function convert_pchar16T_To_pwcharT in kmx_u16

* chore(developer): CheckFilenameConsistency added fclose()

* chore(developer): --

* chore(developer): removed printf (---> started in... / °°-> changed to...)

* chore(developer): AddDeprecatedCode to get CWARN when in kmcomp.exe -s-w is set

* chore(developer): change MakeHashKeyFromFileKey16() to get correct behaviour of CHINT

* chore(developer): change MakeHashKeyFromFileKey16() to get correct behaviour of CHINT

* chore(developer): move converting function to kmx_u16

* chore(developer): add more CERR_ tests

* chore(developer): Not ready yet: changes in CheckFilenameConsistency to use on non-windows

* chore(developer): Not ready yet: changes in CheckFilenameConsistency to use on non-windows

* chore(developer): copied contents of 3 files of commit fcfc6c698a to here to achieve state of fcfc6c698a

* chore(developer): copied contents of 3 files of commit fcfc6c698a to here to achieve state of fcfc6c698a

* chore(developer): move/rename converting functions

* chore(developer) : update .gitignore, remove converting function

* chore(developer) : update gitignore; sort code in CheckFilenameConsistency::CheckFilenameConsistency()

* chore(developer) : change Version number 15.0.270 -> 17.0.51

* chore(developer): change #includes, path in meson.build

* chore(developer): new function Open_File() to open files on windows and non-windows platforms

* chore(developer): create Function to replace fopen/_wfsopen

* chore(developer): exclude kmx, kvk from git

* chore(developer): exclude kmx, kvk from git

* chore(developer): change #includes to include from kmcmplib

* chore(developer): remove stray fopen

* chore(developer): include Open_File() (as comment for now)

* chore(developer): include Open_File() (as comment for now)

* chore(developer): remove path to kmcmpdll in include_directories

* chore(developer): use functin Open_File()

* chore(developer): remove #ifdef for '\\' and '/' in both IsRelativePath; remove Function Open_File again

* chore(developer): new fun  u16rchr_LinWin, strrchr_LinWin to exchange #ifndef for '\\' and '/'

* chore(developer): implement u16rchr_LinWin and strrchr_LinWin

* chore(developer): implement Open_File()

* chore(developer): new MakeHashKeyFromFileKey, remove comments for Open_File, & *LinWin-functions, delete 1 converting function

* chore(developer): tidy up comments

---------

Co-authored-by: Marc Durdin <marc@durdin.net>
2023-03-01 20:00:04 +11:00
Marc Durdin
2de745b3df feat(developer): add 'Use Legacy Compiler' option to TIKE 2023-02-28 15:55:38 +07:00
Steven R. Loomis
712063d752
Merge pull request #8316 from keymanapp/fix/developer-8069/todo-epic-ldml
fix(developer): ldml TODO-LDML cleanup 🙀
2023-02-27 13:16:57 -06:00
Marc Durdin
6846a6bb70 feat(developer): add -use-legacy-compiler flag to kmcomp 2023-02-27 16:00:02 +07:00
Marc Durdin
0b7b22999e
Merge pull request #8326 from keymanapp/feat/developer/2761-kmc-build-project-2.0
feat(developer): Add support for building a folder to kmc 🙀
2023-02-27 19:56:42 +11:00
Marc Durdin
05d5c6f1f6
Merge pull request #8320 from keymanapp/feat/developer/5283-kmc-kpj-support
feat(developer): kmc .kpj support 🙀
2023-02-27 19:56:33 +11:00
Marc Durdin
6c87b49996
Merge pull request #8317 from keymanapp/feat/common/5283-kpj-schema-and-reader
feat(common): add .kpj typescript reader and schema 🙀
2023-02-27 19:56:24 +11:00
Marc Durdin
89f26e1f9b
Merge pull request #8311 from keymanapp/feat/developer/5283-kmc-model-ts-support
feat(developer): kmc compile .model.ts file 🙀
2023-02-27 19:56:13 +11:00
Marc Durdin
d541c3b990
Merge pull request #8310 from keymanapp/feat/developer/5283-kmc-kps-support
feat(developer): kmc compile .kps files 🙀
2023-02-27 19:56:04 +11:00
Marc Durdin
a9b707d4f1
Merge pull request #8302 from keymanapp/feat/developer/5283-kmc-kmn-wrapper
feat(developer): support build of .kmn from kmc 🙀
2023-02-27 19:55:56 +11:00
Marc Durdin
8f02108579
Update developer/src/kmc/src/commands/buildTestData.ts
Co-authored-by: Steven R. Loomis <srl295@gmail.com>
2023-02-27 17:53:25 +11:00
Marc Durdin
be4f036a3c feat(developer): Add support for building a folder to kmc
Relates to #2761.

kmc now supports passing in a folder and building all Keyman source
files found in the folder, even if a .kpj is not present in the folder.

If a .kpj file exists, `KeymanDeveloperProject/Options/Version` will be
checked to determine behaviour. If this field is missing or "1.0", then
the classical behaviour persists.

If the Version field is "2.0", the `Files` key is ignored, and .kmn,
.xml, .model.ts, and .kps files under `sourcePath` are enumerated and
added to the in-memory project for build.

If a .kpj is not present in the folder, a virtual project is constructed
in memory with the following defaults:

* `version`: `2.0`
* `buildPath`: `$PROJECTPATH/build`
* `sourcePath`: `$PROJECTPATH/source`
* `compilerWarningsAsErrors`: `false`
* `warnDeprecatedCode`: `true`
* `checkFilenameConventions`: `true`

It is still considered an error to include both a keyboard and a model
in the same file, but a version 2.0 project will not include the field
determining its type in `Options/ProjectType`; this will instead be
checked during build.

The call to `kmc` can reference a folder or a .kpj file. The .kpj file
must have the same basename as the folder name (e.g.
`khmer_angkor/khmer_angkor.kpj`).

WARNING: this does not add support for 2.0 projects to Keyman Developer;
it is likely that opening a version 2.0 project in Developer will
currently corrupt it.

WARNING: it is not anticipated that kmcomp.exe will be updated to
support version 2.0 projects -- rather kmcomp.exe will be removed before
release of v17, and replaced with kmc.
2023-02-27 13:11:22 +07:00
Marc Durdin
9af10cdf1d chore(developer): add kpj schema for tests 2023-02-27 08:02:35 +07:00
Marc Durdin
b04489a47b feat(developer): kmc .kpj support
Adds support for building .kpj projects. Required minor tweak to kmcomp
to allow it to build both .kmx and .js from .kmn command line.
Currently, calls kmcomp.exe to build .kmn files, until we support a
cross-platform kmc-kmn C++ project.
2023-02-27 06:14:06 +07:00
Marc Durdin
ec26e73932 chore(developer): skip copying schemas that may not be needed 2023-02-25 15:38:29 +07:00
Marc Durdin
647803ef28 feat(common): add .kpj typescript reader and schema 2023-02-25 11:17:39 +07:00
Steven R. Loomis
77a76a2f54 fix(developer): ldml TODO-LDML cleanup 🙀
- add check for missing flicks
- remove a tray #include

chore(common): ldml TODO-LDML cleanup 🙀  #8069
2023-02-24 18:55:28 -06:00
Sabine
a1cba9eaa5 chore(developer): tidy up comments 2023-02-24 16:55:42 +01:00
Sabine
51c8b6506b chore(developer): new MakeHashKeyFromFileKey, remove comments for Open_File, & *LinWin-functions, delete 1 converting function 2023-02-24 16:33:39 +01:00
Sabine
304a20efad chore(developer): implement Open_File() 2023-02-24 13:11:15 +01:00
Marc Durdin
12a6e404af feat(developer): kmc compile .model.ts file
Adds basic support for compiling .model.ts files using kmc-model to kmc.
2023-02-24 16:18:46 +07:00
Marc Durdin
f6a3364c56 chore(developer): no top-level await 2023-02-24 16:18:27 +07:00
Sabine
67075ad669 chore(developer): implement u16rchr_LinWin and strrchr_LinWin 2023-02-24 10:09:59 +01:00
Marc Durdin
adfe3fac63 feat(developer): kmc compile .kps files
Adds basic support for compiling .kps files using kmc-package to kmc.
2023-02-24 14:29:48 +07:00
Marc Durdin
113918daef chore(developer): use BuildCommandOptions type 2023-02-24 13:57:36 +07:00
Marc Durdin
1be3ab37cc feat(developer): support build of .kmn from kmc
Uses existing kmcomp.exe to build .kmn files. Requires kmc to be in a
subfolder of the Keyman Developer binary folder.
2023-02-24 13:57:35 +07:00
Marc Durdin
91c6cb9f49 chore(developer): use --out-file for build-test-data 2023-02-24 13:54:29 +07:00
Marc Durdin
083be28ac4 chore(developer): fixup mkdir 2023-02-24 13:54:28 +07:00
Marc Durdin
a30fdffd95 feat(developer): kmc build command 2023-02-24 13:54:27 +07:00
Marc Durdin
ca77d0728f chore(developer): remove extraneous commander ref 2023-02-24 13:54:26 +07:00
Marc Durdin
c7c52af3e2 chore(developer): update to commander 10.0.0 2023-02-24 13:54:25 +07:00
Sabine
6d44930a60 chore(developer): new fun u16rchr_LinWin, strrchr_LinWin to exchange #ifndef for '\\' and '/' 2023-02-23 18:27:56 +01:00
Sabine
5597e76d1b chore(developer): remove #ifdef for '\\' and '/' in both IsRelativePath; remove Function Open_File again 2023-02-23 16:14:03 +01:00
Sabine
44d7472b7b chore(developer): use functin Open_File() 2023-02-23 10:52:43 +01:00
Sabine
51a0602596 chore(developer): remove path to kmcmpdll in include_directories 2023-02-23 07:21:08 +01:00
Marc Durdin
38603e8421
Merge pull request #8279 from keymanapp/chore/core/cleanup-meson-build
chore(core): cleanup meson build
2023-02-23 13:28:02 +11:00
Marc Durdin
a2ceb98948 chore: rewrite extra standard build script prolog 2023-02-23 08:09:41 +07:00
Marc Durdin
9eb472579c chore: Merge branch 'feature-ldml' into chore/merge-master-to-feature-ldml-A17S7-2 2023-02-23 08:05:31 +07:00
Marc Durdin
b968bcfdfd
Merge branch 'feature-ldml' into chore/core/cleanup-meson-build 2023-02-23 08:49:56 +11:00