Clarifies the confusing builder.inc.sh / build-utils.sh distinction by
giving the scripts more appropriate names. Most build scripts should use
builder-full.inc.sh; some helper scripts can use builder-basic.inc.sh.
Documented in resources/build/README.md.
Renames:
* resources/build/builder.inc.sh to resources/build/builder-full.inc.sh
* resources/build/build-utils.sh to resources/build/builder-basic.inc.sh
Other changes:
* Moves Android-specific functions out of builder-basic.inc.sh and into
android/build.sh.
* Renames functions in builder-basic.inc.sh
More functions may be moved from builder-basic.inc.sh into utils.inc.sh
or other scripts in the future.
Fixes: #14065
Build-bot: build all
Test-bot: skip
Had a real yak shave this morning with disabling assertions in release
builds in our C/C++ code. It turns out that our unit tests use
`assert()` which we intended to use from `test_assert.h`, but in some
cases `cassert` or `assert.h` had been #included after `test_assert.h`,
overriding our special `assert()` macro. The chain of includes is
somewhat hard to puzzle out -- it's often buried several levels deep.
This meant that a release build would drop all test assertions, meaning
most tests passed, unsurprisingly, as there were no assertions left to
fail ... but some tests failed with crashes because we optimized out
important lines such as `assert(some_important_function())`.
I was quite unhappy with this fragility, so I have opted to rename
`assert()` to `test_assert()` in all of our home-grown C/C++ unit tests,
which further highlighted unit tests which were only using the C/C++
`assert()` and not ours, so then had to figure out which unit test
executables needed to have `test_assert` added, and then ... then ...
discovered a bug in `test_color.h`, where we were #including
`io.h`/`unistd.h` inside a `namespace console_color {}` block, which
just happened to be the first ref to those beautiful headers, and thus
(because `#pragma once`) meant that useful little functions like
`access()` were no longer accessible to us in the global namespace.
I have also audited Every Single Call to `assert()` to verify that we do
not do Important Work inside the parentheses, and, apart from those
offending unit tests, now resolved with `test_assert()`, it looks like
all is good.
I would like to present one very well-shaved yak in this commit.
Fixes: #12619
Moves variable declaration for `bool enabled` from .h to .cpp and fixes
up builds. Fixes `#include` in ldml_test_source.cpp which was inside a
namespace.
Fixes: #10426
feat(mac): include km_types and u16
feat(mac): include filesystem and some kmx
feat(mac): include old functions in keymap
feat(mac): include kmx_file, mc_kmxfile
feat(mac): include deadkey, filesystem, km_types, import_rules,mc_kmxfile
feat(mac): include functions in mcompile.cpp/h
feat(mac): #includes out to be used
Note: there is a bit of potential confusion about the difference between
/resources/builder.inc.sh (the full implementation for builder scripts),
and /resources/build/builder.inc.sh (the source script that builder
scripts should always use).
This allows us to make assumptions that will always be true for builder
scripts that may not be true for other scripts, such as setting base
folder.
Fixes#11324.
* Always `cd "$THIS_SCRIPT_PATH"`
* Remove unnecessary `cd` from all build.sh
* Remove unnecessary `set -eu` from all build.sh (and `# set -x`)
* Replace old build-utils.sh incantation in a few build.sh scripts
Fixes#9541.
Also adds version 16.0 support to the kmcmplib compiler constants, and
unit tests for both versions. Does not add any support for automatic
version feature detection, because that forces an inverted dependency on
the touch layout compilation phase (done in kmc-kmw), which would be a
significant refactor. This may be something we need to support in the
future.
When we read .kmx files, they have no alignment guarantees, so we need
to tell the compiler to generate unaligned-safe code for accesses to
`COMP_` structure members, because typically we point into the buffer
and read at any offset.
A .kmx file will generally be 2-byte aligned, as there are no structures
or data types with smaller than 2 byte widths. There is no requirement
that this be the case per the .kmx spec, though. So we use a 1 byte
alignment attribute for the compiler, so it will generate safe code when
reading these structs.
Note that we are assuming that `COMP_KEYBOARD` is aligned because it is
always the start of the file, so will be at the start of any buffer
which will automatically be aligned correctly.
We should probably review some of our design decisions and structs for
KMXPlus given this.
The &displayMap system store adds support for a mapping file that remaps
the On Screen Keyboard files -- .kvks and .keyman-touch-layout, at
compile time, to new ranges. This is specifically intended to resolve a
limitation of unattached marks and diacritics which display
inconsistently across platforms; this is described in more detail in
issue #9031.
In order to allow us to reference the padding in the FILE_KEY structure,
for reproducible and cross-platform consistent builds, turn the padding
into an `_reserved` (must always be zero) field.
Changes to compiler to utilise this will be in a follow-up commit.
* get keymans version
Merge branch 'master' of https://github.com/keymanapp/keyman into refactor/developer/6026-kmcompx-compiler-cross-platform-part-2-NEW_VERSION
# Conflicts:
# common/core/desktop/src/kmx/kmx_context.h
# common/core/desktop/src/kmx/kmx_processor.cpp
# common/windows/cpp/include/keymanversion.h
# common/windows/cpp/include/kkmnkbd/Compfile.h
# common/windows/cpp/include/kmtip_guids.h
# common/windows/cpp/src/kmtip_guids.cpp
# core/src/kmx/kmx_conversion.cpp
# core/src/kmx/kmx_conversion.h
# core/src/kmx/kmx_file.cpp
# core/src/kmx/kmx_processevent.cpp
# core/src/kmx/kmx_processor.hpp
# core/src/kmx/kmx_xstring.cpp
# developer/js/.gitignore
# developer/js/package-lock.json
# developer/js/tests/tsconfig.json
# developer/src/kmcmpdll/CasedKeys.cpp
# developer/src/kmcmpdll/CheckFilenameConsistency.cpp
# developer/src/kmcmpdll/CheckFilenameConsistency.h
# developer/src/kmcmpdll/CheckForDuplicates.cpp
# developer/src/kmcmpdll/CheckForDuplicates.h
# developer/src/kmcmpdll/CheckNCapsConsistency.cpp
# developer/src/kmcmpdll/CheckNCapsConsistency.h
# developer/src/kmcmpdll/Compiler.cpp
# developer/src/kmcmpdll/DeprecationChecks.cpp
# developer/src/kmcmpdll/NamedCodeConstants.cpp
# developer/src/kmcmpdll/UnreachableRules.cpp
# developer/src/kmcmpdll/UnreachableRules.h
# developer/src/kmcmpdll/compfile.h
# developer/src/kmcmpdll/kcframe/kcframe.cpp
# developer/src/kmcmpdll/kcframe/kcframe.vcxproj
# developer/src/kmcmpdll/kcframe/kcframe.vcxproj.filters
# developer/src/kmcmpdll/kmcmpdll.vcxproj
# developer/src/kmcmpdll/kmcmpdll.vcxproj.filters
# developer/src/kmcmpdll/version.rc
# developer/src/kmcmpdll/versioning.cpp
# developer/src/kmlmc/bundle.sh
# developer/src/kmlmc/tests/test-join-word-breaker.ts
# developer/src/kmlmc/tests/test-override-script-defaults.ts
# developer/src/server/tsconfig.json
# windows/src/developer/kmcmpdll/Makefile
# windows/src/developer/kmcmpdll/vkeys.h
# windows/src/engine/keyman32/keymanengine.h
# windows/src/global/inc/Vkeys.h
# windows/src/global/inc/rc4.h
# windows/src/global/vc/rc4.cpp
* changes in meson + include-paths of several files
* chore: remove common/core
* chore: remove unnecessary files
* chore: move headers and refs
* refactor(developer): fixup path to tests
* refactor(developer): fixup path to tests
tidied up +removed unnecessary files
checked contents of files+included a few new functions needed for myVersion
changed datatypes in CheckForDuplicates
tidy up more paths
CheckFileConsistency: not able to fully adapt for use of char16_t on non windows p.
-> used char16_t + commented out some AddWarning().
* sorted out some #includes
renamed some functions to reduce ambiguity
* fix(developer): solved linker errors
* chore(developer): remove some comments
* chore(developer): introduced KMX_UCHAR
* chore(developer): restored whitespace, comments
* chore(developer): tidied up vkeys.H / .cpp
* chore(developer): remove keyboardprocessor...
* chore(common): refactor shared types into km_types.h
* chore(core): remove duplicate types
* chore(core): move kmx_file.h to common
* chore(developper): removed typedefs of kmcompcx.h
* chore(developer): removed kmcompx/include/kmx_file.h
* chore(developer): chanched include of pch.h
* chore(developer): removed files from kmcompx/include/
* chore(developer): removed commented sections ( ** old **) if files
* chore(developer): removed doubled/overwritten functions
* chore(developer): removed doubled/overwritten functions
* chore(developer): changed #includes ".." -> <..>
* chore(developer): changed #includes ".." -> <..>
* chore(developer): enable tests for keyboards repo for kmcompx
* chore(developer): new functions e.g. string_from_wstring...
* chore(developer): changed printf->u16sprintf
* chore(developer): fopen
* chore(developer): fixed errors on running kmcompxtest
* chore(developer): changef u16fmt
* chore(developer): fixup u16ncat
* chore(developer): added return Msgs for different types of errors while compiling
* chore(developer): tests using RTL run Ok ( changed RTu->RTL)
* chore(developer): replaced u16ncmp with u16nicmp at most places
* chore(developer): changed Version number for tests 16.046 to 15.0265
* chore(developer): mostly whitespace changes
* chore(developer): tidied up code (not finished yet)
* chore(developer): removed #pragma once from wrong places, edited #includes
* chore(developer): removed #pragma once from wrong places, edited #includes
* chore(developer): CheckFileNameConsistency changed to take KMX_WCHART const* (= wchar_t* const); used in compiler.cpp now
* chore(developer): excluded some tests from kmcompxtest (use get_test_source_exclude_some_kmn.bat now)
* chore(developer): Win-functions in FileNameConsistency
* chore(developer): Win-functions in FileNameConsistency
* chore(developer): CheckFNConsuistency: called u16printf with f´different par.
* chore(developer): tidied up code
* chore(developer): tidied up code
* chore(developer): tiedied up code (compiler.cpp)
* chore(developer): tiedied up code
* chore(developer): tidied up code
* chore(developer): removed all commentaries _S2
* chore(developer): tidied up code
* chore(developer): tidied up code
* chore(developer): tidied up code
* chore(developer): removed several Windows-specific functions
* chore(developer): removed unused datatype typedefs
* chore(developer): new GetCompilerErrorString() used in AddCompileMessage()
* chore(developer): new GetCompilerErrorString() used in AddCompileMessage()
* chore(developer): removed CWARN/CHINT out of CompMsg
* chore(developer): adapted kmcompxtest to test kmn-files that have errors included
* chore(developer): changed gitignore
* chore(developer): changed gitignore
* chore(developer): changed Version Number fot Tests 15.0.265.0 -> 15.0.270.0
* chore(developer): added tests for kmcompx
* chore(developer): move tests (for ErrorMessages) to common/test/keyboards
* chore(developer): move CERR-tests into test/keyboards/invalid
* chore(developer): change mostly whitespace changes
* chore(developer): change mostly whitespace changes
* chore(developer): delete files of keyboards/kmcompx_tests
* chore(developer): include check for correct Filename + rename var + change whitespace changes
* chore(developer): revert changes in AddCompileMessage, tidy up CreateTempFile
* chore(developer): remove python from meson.build
* chore(developer): change ErrExtra, change CompileMsg, change _finddata in FileExists
* chore(developer): check for '_' after Errornumber in Filename
* chore(developer): re-include AddCompilerVersionStore in Compiler.cpp
* chore(developer): add Message CERR_InvalidCharacter to UTF16TempFromUTF8() ; change #include <> -> ""
* chore(developer): add some more keyboardtests
* chore(developer): add some more keyboardtests
* chore(developer): change remaining old datatype-names to new names ( e.g. char -> KMX_CHAR )
* chore(developer): edit + add CERR_tests
* chore(developer): edit + add CERR_tests
* chore(developer): use char16_t in CheckFileNameConsistency
* chore(developer): comments: converting functions in kmx_u16
* chore: reorganize
* chore: commit kmcompx-files
* chore: use ifdef for vkeys
* chore: copy files back
* chore: more cleanup
* chore: meson build for kmcmplib
* chore: tweak unit test, error messages, and paths
* chore(developer): make kmcmplib a static library
* chore(developer): start kmcmpdll link to kmcmplib
* chore(developer): add build.sh for kmcmplib
* chore(developer): link kmcmpdll to kmcmplib
* chore(common): avoid building 'invalid' shared keyboards
* chore(developer): include BEGIN_LINE (repeated Begin)
* chore(developer): add more CERR_tests and move some to other folder, edit readme
* chore(developer): addCompilerMsg for RepeatedBegin
* chore(developer):repeatedBEgin plus rename functions to prevent same names in kmcmpdll and kmcmplib
* chore(developer): add flag flag_use_kmcompx as switch to use old code (kmcmpdll) or new code kmcmplib
* chore(developer): introduce namespace kmcmp instead of different filenames
* chore(developer): more changes for using namespaces
* chore(developer): solve merge conflicts in namespaces
* chore(developer): use namespaces in kmcmplib (except for in dllexport-functions)
* chore(developer): remove some unnecessary CERR_ tests
* chore(developer): remove CERR_CallIsProfessionalFeature, CERR_IncludeCodesIsProfessionalFeature, CERR_MnemonicLayoutIsProfessionalFeature
* chore(developer): note about func.call_js; change all "VESION x.y " to "VESION x.y or higher" ; rename flag for kmcompx to flag_use_new_kmcomp
* chore(developer): use PKMX_DWORD/PKMX_WCHAR instead of LPKMX_DWORD/LPKMX_WCHAR; use PKMX_STR instead of char* in CompilerMessageProc
* chore(developer): remove kmx-file; NamedCodeConstants: rename variables, set const variables for buf sizes, brackets in for-loops
* chore(developer): change include paths ../
* chore(developer): get rid of reopened namespaces of kmcmp
* chore(developer): restored kmcompxtest
* chore(developer): change #include paths ../ ; excange km_kbp_cp with KMX_WCHAR; change some variable names
* chore(developer): use environment var KEYMAN_ROOT in path for tests; remove try-catch from json-validation.cpp
* chore(developer): use environment var KEYMAN_ROOT in path for tests; remove try-catch from json-validation.cpp
* chore(developer): revert change in NamedCodeConstants
* chore(developer): restored kmcompxtest, adapted readme
* chore(developer): use res-fils/version.rc for kmcmplib
* chore(developer): rename some functions/variables
* chore(developer): typos, comments, add function u16rchr()
* chore(developer): typos, comments, add function u16rchr()
* chore(developer): rename Functions, version rc-files
* chore(developer): rename Functions, version rc-files, set flag_use_new_kmcomp to use (old) kmcmpdll
* chore(developer): add BeginLine in kmcmpdll/Compiler.cpp; fread() in kmcmp_CompileKeyboardFileToBuffer
* chore(developer): set flag_use_new_kmcomp to only use kmcmpdll
* chore(developer): set flag_use_new_kmcomp to only use kmcmpdll...and back to kmcmplib
* (chore(developer): set flag_use_new_kmcomp to only use kmcmpdll...and back
* (chore(developer): changes in NamedCodeConstants::reindex() to run hieroglyphic.kmn properly
* chore(developer): changes in UnrechableRules::VerifyUnreachableRules() to run test_6440_unreachable_code..kmn
* chore(developer): remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta
* chore(developer): remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta, CWARN_ OK
* chore(developer): after stash:remove unneccessary files, get ErrorMsg for INCLUDECODES, VISUALKEYBOARD, UnreachableRules, ErrExrta, CWARN_ OK
* chore(developer): replace some #include <> with "" ; rename ErrExtra -> ErrExtraW
* chore(developer): replace some #include <> with "" ; rename ErrExtra -> ErrExtraW
* chore(developer): new converting function convert_pchar16T_To_pwcharT in kmx_u16
* chore(developer): CheckFilenameConsistency added fclose()
* chore(developer): --
* chore(developer): removed printf (---> started in... / °°-> changed to...)
* chore(developer): AddDeprecatedCode to get CWARN when in kmcomp.exe -s-w is set
* chore(developer): change MakeHashKeyFromFileKey16() to get correct behaviour of CHINT
* chore(developer): change MakeHashKeyFromFileKey16() to get correct behaviour of CHINT
* chore(developer): move converting function to kmx_u16
* chore(developer): add more CERR_ tests
* chore(developer): Not ready yet: changes in CheckFilenameConsistency to use on non-windows
* chore(developer): Not ready yet: changes in CheckFilenameConsistency to use on non-windows
* chore(developer): copied contents of 3 files of commit fcfc6c698a to here to achieve state of fcfc6c698a
* chore(developer): copied contents of 3 files of commit fcfc6c698a to here to achieve state of fcfc6c698a
* chore(developer): move/rename converting functions
* chore(developer) : update .gitignore, remove converting function
* chore(developer) : update gitignore; sort code in CheckFilenameConsistency::CheckFilenameConsistency()
* chore(developer) : change Version number 15.0.270 -> 17.0.51
* chore(developer): change #includes, path in meson.build
* chore(developer): new function Open_File() to open files on windows and non-windows platforms
* chore(developer): create Function to replace fopen/_wfsopen
* chore(developer): exclude kmx, kvk from git
* chore(developer): exclude kmx, kvk from git
* chore(developer): change #includes to include from kmcmplib
* chore(developer): remove stray fopen
* chore(developer): include Open_File() (as comment for now)
* chore(developer): include Open_File() (as comment for now)
* chore(developer): remove path to kmcmpdll in include_directories
* chore(developer): use functin Open_File()
* chore(developer): remove #ifdef for '\\' and '/' in both IsRelativePath; remove Function Open_File again
* chore(developer): new fun u16rchr_LinWin, strrchr_LinWin to exchange #ifndef for '\\' and '/'
* chore(developer): implement u16rchr_LinWin and strrchr_LinWin
* chore(developer): implement Open_File()
* chore(developer): new MakeHashKeyFromFileKey, remove comments for Open_File, & *LinWin-functions, delete 1 converting function
* chore(developer): tidy up comments
---------
Co-authored-by: Marc Durdin <marc@durdin.net>
