erabo/spiegel-keyman
1
0
Fork 0
mirror of https://github.com/keymanapp/keyman.git synced 2026-09-28 02:27:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge pull request #11217 from keymanapp/chore/merge-beta-to-master-b17s5

Browse source 
chore(common): Merge beta to master for Sprint B17S5
This commit is contained in:
Darcy Wong 2024-04-12 12:47:39 +07:00 • committed by GitHub
commit 1f3ef92961
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
150 changed files with 3253 additions and 1006 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
.github/workflows/deb-packaging.yml vendored
View file

@ -151,6 +151,16 @@ jobs:
ref: '${{ github.event.client_payload.buildSha }}'
sparse-checkout: '.github/actions/'
binary_packages_unreleased:
name: Build binary packages for next Ubuntu version
needs: sourcepackage
strategy:
fail-fast: true
matrix:
dist: [noble]
runs-on: ubuntu-latest
steps:
- name: Build
uses: ./.github/actions/build-binary-packages
with:

75
HISTORY.md
View file

@ -75,6 +75,81 @@
* chore(common): move to 18.0 alpha (#10713)
* chore: move to 18.0 alpha
## 17.0.306 beta 2024-04-11
* docs(ios): updates iOS app help for 17.0 banner changes (#11200)
* chore(oem/fv): Add fv sguuxs and update keyboard versions (#11198)
## 17.0.305 beta 2024-04-10
* fix(core): skip leading trail surrogate char in km_core_state_context_set_if_needed() (#11169)
* change(web): merges split async method in gesture engine (#11142)
* fix(web): blocks nextLayer for keys quickly typed when multitapping to new layer when final tap is held (#11189)
* refactor(web): OSK spacebar-label updates now managed by layer object (#11175)
## 17.0.304 beta 2024-04-09
* fix(android): atomically updates selection with text (#11188)
* fix(web): fix crash in nearest-key row lookup when touch moves out-of-bounds (#11178)
## 17.0.303 beta 2024-04-05
* fix(windows): decode uri for Package ID and filename (#11152)
* fix(common/models): suggestion stability after multiple whitespaces (#11164)
## 17.0.302 beta 2024-04-04
* fix(mac): load only 80 characters from context when processing keystrokes (#11141)
## 17.0.301 beta 2024-04-03
* feat(core): support modifiers=other (#11118)
* chore(core): dx better err message on embedded test vkeys (#11119)
* fix(web): key preview stickiness 🪠 (#10778)
* fix(web): early gesture-match abort when unable to extend existing gestures 🪠 (#10836)
* fix(web): infinite model-match replacement looping 🪠 (#10838)
* fix(web): proper gesture-match sequencing 🪠 (#10840)
* change(web): input-event sequentialization 🪠 (#10843)
* fix(web): proper linkage of sources to events 🪠 (#10960)
* fix(developer): handle buffer boundaries in four cases (#11137)
* chore(linux): Build packages for next Ubuntu version separately (#11153)
* fix(common): upgrade sentry-cli to 2.31.0 (#11151)
* fix(android/app): Track previous device orientation for SystemKeyboard (#11134)
* change(web): reworks nearest-key detection to avoid layout reflow (#11129)
## 17.0.300 beta 2024-04-02
* change(web): keyboard swaps keep original keyboards active until fully ready (#11108)
* fix(android/engine): Swap selection range if reversed (#11127)
* test(developer): keyboard info compiler unit tests (#11000)
## 17.0.299 beta 2024-04-01
* fix(ios): address crash by reading full code point rather than code unit when trimming initial directional-mark (#11113)
* fix(mac): delete correct number of characters from current context when processing BMP or SMP deletes (#11086)
* feat(developer): disallow stray dollarsign in from pattern (#11117)
## 17.0.298 beta 2024-03-29
* chore(linux): Update debian changelog (#11096)
* fix(web): prevent layer switch key from erasing selection (#11032)
* fix(developer): prevent error when scrolling touch layout editor with no selected key (#11109)
* fix(common): make `isEmptyTransform` return true if passed a nullish transform (#11110)
## 17.0.297 beta 2024-03-28
* fix(common): properly handle illegal UnicodeSets to prevent crash in kmc-ldml compiler (#11065)
* fix(core,developer): variable/marker substitution in sets and strings (#11059)
* fix(developer): in ldml compiler, generate compiler error if `from=` regex matches empty string (#11070)
* fix(core): calculate offset correctly when replacing marker in transform (fixes crash) (#11071)
* feat(developer): support comma in modifiers (#11075)
* fix(core): actions_normalize() length and dead store fix (#11100)
* chore(core): optimize ldml_event_state::emit_difference() when no diff (#11094)
* fix(ios): bad initial in-app layout, delayed banner, deprecated banner toggle (#10929)
* feat(developer/compilers): better unit test for suggestion accessibility (#11085)
* fix(core): fix pointer math in actions_normalize() (#11101)
## 17.0.296 beta 2024-03-27
* fix(developer): in model compiler, give correct key to shorter prefix words when a longer, higher-frequency word is also present (#11074)

15
android/KMAPro/kMAPro/src/main/java/com/keyman/android/SystemKeyboard.java
View file

@ -42,6 +42,7 @@ public class SystemKeyboard extends InputMethodService implements OnKeyboardEven
private static View inputView = null;
private static ExtractedText exText = null;
private KMHardwareKeyboardInterpreter interpreter = null;
private int lastOrientation = Configuration.ORIENTATION_UNDEFINED;
private static final String TAG = "SystemKeyboard";
@ -129,7 +130,7 @@ public class SystemKeyboard extends InputMethodService implements OnKeyboardEven
@Override
public void onUpdateSelection(int oldSelStart, int oldSelEnd, int newSelStart, int newSelEnd, int candidatesStart, int candidatesEnd) {
super.onUpdateSelection(oldSelStart, oldSelEnd, newSelStart, newSelEnd, candidatesStart, candidatesEnd);
KMManager.updateSelectionRange(KMManager.KeyboardType.KEYBOARD_TYPE_SYSTEM, newSelStart, newSelEnd);
KMManager.updateSelectionRange(KMManager.KeyboardType.KEYBOARD_TYPE_SYSTEM);
}
/**
@ -169,9 +170,7 @@ public class SystemKeyboard extends InputMethodService implements OnKeyboardEven
ExtractedText icText = ic.getExtractedText(new ExtractedTextRequest(), 0);
if (icText != null) {
boolean didUpdateText = KMManager.updateText(KeyboardType.KEYBOARD_TYPE_SYSTEM, icText.text.toString());
int selStart = icText.startOffset + icText.selectionStart;
int selEnd = icText.startOffset + icText.selectionEnd;
boolean didUpdateSelection = KMManager.updateSelectionRange(KeyboardType.KEYBOARD_TYPE_SYSTEM, selStart, selEnd);
boolean didUpdateSelection = KMManager.updateSelectionRange(KeyboardType.KEYBOARD_TYPE_SYSTEM);
if (!didUpdateText || !didUpdateSelection)
exText = icText;
}
@ -197,7 +196,10 @@ public class SystemKeyboard extends InputMethodService implements OnKeyboardEven
@Override
public void onConfigurationChanged(Configuration newConfig) {
super.onConfigurationChanged(newConfig);
KMManager.onConfigurationChanged(newConfig);
if (newConfig.orientation != lastOrientation) {
lastOrientation = newConfig.orientation;
KMManager.onConfigurationChanged(newConfig);
}
}
@Override
@ -220,8 +222,9 @@ public class SystemKeyboard extends InputMethodService implements OnKeyboardEven
wm.getDefaultDisplay().getSize(size);
int inputViewHeight = 0;
if (inputView != null)
if (inputView != null) {
inputViewHeight = inputView.getHeight();
}
int bannerHeight = KMManager.getBannerHeight(this);
int kbHeight = KMManager.getKeyboardHeight(this);

14
android/KMEA/app/src/main/java/com/keyman/engine/KMKeyboard.java
View file

@ -163,7 +163,7 @@ final class KMKeyboard extends WebView {
return result;
}
protected boolean updateSelectionRange(int selStart, int selEnd) {
protected boolean updateSelectionRange() {
boolean result = false;
InputConnection ic = KMManager.getInputConnection(this.keyboardType);
if (ic != null) {
@ -175,6 +175,16 @@ final class KMKeyboard extends WebView {
String rawText = icText.text.toString();
updateText(rawText.toString());
int selStart = icText.selectionStart;
int selEnd = icText.selectionEnd;
int selMin = selStart, selMax = selEnd;
if (selStart > selEnd) {
// Selection is reversed so "swap"
selMin = selEnd;
selMax = selStart;
}
/*
The values of selStart & selEnd provided by the system are in code units,
not code-points. We need to account for surrogate pairs here.
@ -193,8 +203,8 @@ final class KMKeyboard extends WebView {
selStart -= pairsAtStart;
selEnd -= (pairsAtStart + pairsSelected);
this.loadJavascript(KMString.format("updateKMSelectionRange(%d,%d)", selStart, selEnd));
}
this.loadJavascript(KMString.format("updateKMSelectionRange(%d,%d)", selStart, selEnd));
result = true;
return result;

27
android/KMEA/app/src/main/java/com/keyman/engine/KMManager.java
View file

@ -2137,17 +2137,40 @@ public final class KMManager {
return result;
}
/**
* Updates the active range for selected text.
* @deprecated
* This method no longer needs the `selStart` and `selEnd` parameters.
* <p>Use {@link KMManager#updateSelectionRange(KeyboardType)} instead.</p>
*
* @param kbType A value indicating if this request is for the in-app keyboard or the system keyboard
* @param selStart (deprecated) the start index for the range
* @param selEnd (deprecated) the end index for the selected range
* @return
*/
@Deprecated
public static boolean updateSelectionRange(KeyboardType kbType, int selStart, int selEnd) {
return updateSelectionRange(kbType);
}
/**
* Performs a synchronization check for the active range for selected text,
* ensuring it matches the text-editor's current state.
* @param kbType A value indicating if this request is for the in-app or system keyboard.
* @return
*/
public static boolean updateSelectionRange(KeyboardType kbType) {
boolean result = false;
if (kbType == KeyboardType.KEYBOARD_TYPE_INAPP) {
if (isKeyboardLoaded(KeyboardType.KEYBOARD_TYPE_INAPP) && !InAppKeyboard.shouldIgnoreSelectionChange()) {
result = InAppKeyboard.updateSelectionRange(selStart, selEnd);
result = InAppKeyboard.updateSelectionRange();
}
InAppKeyboard.setShouldIgnoreSelectionChange(false);
} else if (kbType == KeyboardType.KEYBOARD_TYPE_SYSTEM) {
if (isKeyboardLoaded(KeyboardType.KEYBOARD_TYPE_SYSTEM) && !SystemKeyboard.shouldIgnoreSelectionChange()) {
result = SystemKeyboard.updateSelectionRange(selStart, selEnd);
result = SystemKeyboard.updateSelectionRange();
}
SystemKeyboard.setShouldIgnoreSelectionChange(false);

6
android/KMEA/app/src/main/java/com/keyman/engine/KMTextView.java
View file

@ -63,10 +63,8 @@ public final class KMTextView extends AppCompatEditText {
*/
public static void updateTextContext() {
KMTextView textView = (KMTextView) activeView;
int selStart = textView.getSelectionStart();
int selEnd = textView.getSelectionEnd();
KMManager.updateText(KeyboardType.KEYBOARD_TYPE_INAPP, textView.getText().toString());
if (KMManager.updateSelectionRange(KeyboardType.KEYBOARD_TYPE_INAPP, selStart, selEnd)) {
if (KMManager.updateSelectionRange(KeyboardType.KEYBOARD_TYPE_INAPP)) {
KMManager.resetContext(KeyboardType.KEYBOARD_TYPE_INAPP);
}
}
@ -167,7 +165,7 @@ public final class KMTextView extends AppCompatEditText {
protected void onSelectionChanged(int selStart, int selEnd) {
super.onSelectionChanged(selStart, selEnd);
if (activeView != null && activeView.equals(this)) {
if (KMManager.updateSelectionRange(KMManager.KeyboardType.KEYBOARD_TYPE_INAPP, selStart, selEnd)) {
if (KMManager.updateSelectionRange(KMManager.KeyboardType.KEYBOARD_TYPE_INAPP)) {
KMManager.resetContext(KeyboardType.KEYBOARD_TYPE_INAPP);
}
}

3
common/include/kmx_file.h
View file

@ -302,6 +302,9 @@ namespace kmx {
#define K_MODIFIERFLAG 0x007F
#define K_NOTMODIFIERFLAG 0xFF00 // I4548
// Note: OTHER_MODIFIER = 0x10000, used by KMX+ for the
// other modifier flag in layers, > 16 bit so not available here.
// See keys_mod_other in keyman_core_ldml.ts
struct COMP_STORE {
KMX_DWORD_unaligned dwSystemID;

98
common/web/gesture-recognizer/src/engine/headless/asyncClosureDispatchQueue.ts Normal file
View file

@ -0,0 +1,98 @@
import { timedPromise } from "@keymanapp/web-utils";
import { reportError } from "../reportError.js";
export type QueueClosure = () => (Promise<any> | void);
/**
This class is modeled somewhat after Swift's `DispatchQueue` class, but with
the twist that each closure may return a `Promise` (in Swift: a `Future`) to
lock out further closure processing until the `Promise` resolves.
*/
export class AsyncClosureDispatchQueue {
private queue: QueueClosure[];
private waitLock: Promise<any>;
private defaultWaitFactory: () => Promise<any>;
/**
*
* @param defaultWaitFactory A factory returning Promises to use for default
* delays between tasks. If not specified, Promises corresponding to
* setTimeout(0) will be used, allowing the microqueue task to flush between
* tasks.
*/
constructor(defaultWaitFactory?: () => Promise<any>) {
// We only need to trigger events if the queue has no prior entries and there isn't an
// active wait-lock; for the latter, we'll auto-trigger the next function when it unlocks.
this.queue = [];
this.defaultWaitFactory = defaultWaitFactory || (() => { return timedPromise(0) });
}
get defaultWait() {
return this.defaultWaitFactory();
}
get ready() {
return this.queue.length == 0 && !this.waitLock;
}
private async triggerNextClosure() {
if(this.queue.length == 0) {
return;
}
const functor = this.queue.shift();
// A stand-in so that `ready` doesn't report true while the closure runs.
this.waitLock = Promise.resolve();
/*
It is imperative that any errors triggered by the functor do not prevent this method from setting
the wait lock that will trigger the following event (if it exists). Failure to do so will
result in all further queued closures never getting the opportunity to run!
*/
let result: undefined | Promise<any>;
try {
// Is either undefined (return type: void) or is a Promise.
result = functor() as undefined | Promise<any>;
/* c8 ignore start */
} catch (err) {
reportError('Error from queued closure', err);
}
/* c8 ignore end */
/*
Replace the stand-in with the _true_ post-closure wait.
If the closure returns a Promise, the implication is that the further processing of queued
functions should be blocked until that Promise is fulfilled.
If not, we just add a default delay.
*/
result = result ?? this.defaultWaitFactory();
this.waitLock = result;
try {
await result;
} catch(err) {
reportError('Async error from queued closure', err);
}
this.waitLock = null;
// if queue is length zero, auto-returns.
this.triggerNextClosure();
}
runAsync(closure: QueueClosure) {
// Check before putting the closure on the internal queue; the check is based in part
// upon the existing queue length.
const isReady = this.ready;
this.queue.push(closure);
// If `!isReady`, the next closure will automatically be triggered when possible.
if(isReady) {
this.triggerNextClosure();
}
}
}

73
common/web/gesture-recognizer/src/engine/headless/gestures/matchers/gestureMatcher.ts
View file

@ -157,14 +157,14 @@ export class GestureMatcher<Type, StateToken = any> implements PredecessorMatch<
return;
case 'full':
contact = srcContact.constructSubview(false, true);
this.addContactInternal(contact, srcContact.path.stats);
this.addContactInternal(contact, srcContact.path.stats, true);
continue;
case 'partial':
preserveBaseItem = true;
// Intentional fall-through
case 'chop':
contact = srcContact.constructSubview(true, preserveBaseItem);
this.addContactInternal(contact, srcContact.path.stats);
this.addContactInternal(contact, srcContact.path.stats, true);
break;
}
}
@ -193,10 +193,16 @@ export class GestureMatcher<Type, StateToken = any> implements PredecessorMatch<
// Easy peasy - resolutions only need & have the one defined action type.
action = this.model.resolutionAction;
} else {
// Some gesture types may wish to restart with a new base item if they fail due to
// it changing during its lifetime or due to characteristics of the contact-point's
// path.
if(this.model.rejectionActions?.[cause]) {
/*
Some gesture types may wish to restart with a new base item if they fail due to
it changing during its lifetime or due to characteristics of the contact-point's
path.
If a gesture model match is outright-cancelled, matcher restarts should be completely
blocked. One notable reason: if a model-match is _immediately_ cancelled due to
initial conditions, reattempting it can cause an infinite (async) loop!
*/
if(cause != 'cancelled' && this.model.rejectionActions?.[cause]) {
action = this.model.rejectionActions[cause];
action.item = 'none';
}
@ -361,7 +367,7 @@ export class GestureMatcher<Type, StateToken = any> implements PredecessorMatch<
return this._result;
}
private addContactInternal(simpleSource: GestureSourceSubview<Type>, basePathStats: CumulativePathStats<Type>) {
private addContactInternal(simpleSource: GestureSourceSubview<Type>, basePathStats: CumulativePathStats<Type>, whileInitializing?: boolean) {
// The number of already-active contacts tracked for this gesture
const existingContacts = this.pathMatchers.length;
@ -437,6 +443,7 @@ export class GestureMatcher<Type, StateToken = any> implements PredecessorMatch<
}
}
// Check that initial "item" and "state" properties are legal for this type of gesture.
if(contactSpec.model.allowsInitialState) {
const initialStateCheck = contactSpec.model.allowsInitialState(
simpleSource.currentSample,
@ -450,13 +457,59 @@ export class GestureMatcher<Type, StateToken = any> implements PredecessorMatch<
// pathMatcher for a source that failed to meet initial conditions.
this.pathMatchers.pop();
this.finalize(false, 'path');
/*
To prevent any further retries for the model (via rejectionActions), we list the
cause as 'cancelled'. 'Cancelled' match attempts will never be retried, and we
wish to prevent an infinite (async) loop from retrying something we know will
auto-cancel. (That loop would automatically end upon a different model's match
or upon all possible models failing to match at the same time, but it's still
far from ideal.)
The rejection-action mechanism in MatcherSelector's `replacer` method (refer to
https://github.com/keymanapp/keyman/blob/be867604e4b2650a60e69dc6bbe0b6115315efff/common/web/gesture-recognizer/src/engine/headless/gestures/matchers/matcherSelector.ts#L559-L575)
already blocks paths that are rejected synchronously by this method. Use of
'cancelled' is thus not necessary for avoiding the loop-scenario, but it does
add an extra layer of protection. Also, it's more explicit about the fact that
we _are_ permanently cancelling any and all future attempts to match against
it in the future for the affected `GestureSource`(s).
If we weren't using 'cancelled', 'item' would correspond best with a rejection
here, as the decision is made due to a validation check against the initial item.
*/
this.finalize(false, 'cancelled');
}
}
contactModel.update();
// Now that we've done the initial-state check, we can check for instantly-matching path models.
/*
Now that we've done the initial-state check, we can check for instantly-matching and
instantly-rejecting path models... particularly from from causes other than initial-item
and state, such as rejection due to an extra touch.
KMW example: longpresses cancel when a new touch comes in during the longpress timer;
they should never become valid again for that base touch.
*/
const result = contactModel.update();
if(result?.type == 'reject') {
/*
Refer to the earlier comment in this method re: use of 'cancelled'; we
need to prevent any and all further attempts to match against this model
We'd instantly reject it anyway due to its rejected initial state.
Failing to do so can cause an infinite async loop.
If we weren't using 'cancelled', 'path' would correspond best with a
rejection here, as the decision is made due to the GestureSource's
current path being rejected by one of the `PathModel`s comprising the
`GestureModel`.
If the model's already been initialized, it's possible that a _new_
incoming touch needs special handling. We'll allow one reset. In the
case that it would try to restart itself, the restarted model will
instantly fail and thus cancel.
*/
this.finalize(false, whileInitializing ? 'cancelled' : 'path');
}
// Standard path: trigger either resolution or rejection when the contact model signals either.
contactModel.promise.then((resolution) => {
this.finalize(resolution.type == 'resolve', resolution.cause);
});

46
common/web/gesture-recognizer/src/engine/headless/gestures/matchers/matcherSelector.ts
View file

@ -15,6 +15,15 @@ interface GestureSourceTracker<Type, StateToken> {
*/
source: GestureSource<Type>;
matchPromise: ManagedPromise<MatcherSelection<Type, StateToken>>;
/**
* Set to `true` during the timeout period needed to complete existing trackers &
* initialize new ones. Once that process is complete, set to false.
*
* This is needed to ensure that failure to extend an existing gesture doesn't
* result in outright selection-failure before attempting to match as a
* newly-started gesture.
*/
preserve: boolean;
}
export interface MatcherSelection<Type, StateToken = any> {
@ -225,10 +234,22 @@ export class MatcherSelector<Type, StateToken = any> extends EventEmitter<EventM
const unmatchedSource = sourceNotYetStaged ? source : null;
const priorMatcher = sourceNotYetStaged ? null: source;
// matchGesture calls should be queued and act atomically, in sequence.
if(this.pendingMatchSetup) {
const parentLockPromise = this.pendingMatchSetup;
const childLock = new ManagedPromise<void>();
this.pendingMatchSetup = childLock.corePromise;
// If a prior call is still waiting on the `await` below, wait for it to clear
// entirely before proceeding; there could be effects for how the next part below is processed.
await this.pendingMatchSetup;
await parentLockPromise;
if(this.pendingMatchSetup == childLock.corePromise) {
this.pendingMatchSetup = null;
}
childLock.resolve(); // allow the next matchGesture call through.
}
if(sourceNotYetStaged) {
@ -257,7 +278,8 @@ export class MatcherSelector<Type, StateToken = any> extends EventEmitter<EventM
// Promises only resolve once, after all - once called, a "selection" has been made.
const sourceSelectors: GestureSourceTracker<Type, StateToken> = {
source: src,
matchPromise: matchPromise
matchPromise: matchPromise,
preserve: true
};
this._sourceSelector.push(sourceSelectors);
@ -303,14 +325,20 @@ export class MatcherSelector<Type, StateToken = any> extends EventEmitter<EventM
* Reference: https://javascript.info/event-loop
*/
const pendingMatchGesture = new ManagedPromise<void>();
this.pendingMatchSetup = pendingMatchGesture.corePromise;
const matchingLock = new ManagedPromise<void>();
this.pendingMatchSetup = matchingLock.corePromise;
await timedPromise(0);
// A second one, in case of a deferred modipress completion (via awaitNested)
// (which itself needs a macroqueue wait)
await timedPromise(0);
this.pendingMatchSetup = null;
pendingMatchGesture.resolve();
// Only clear the promise if no extra entries were added to the implied `matchGesture` queue.
if(this.pendingMatchSetup == matchingLock.corePromise) {
this.pendingMatchSetup = null;
}
matchingLock.resolve();
// stateToken may have shifted by the time we regain control here.
const incomingStateToken = this.stateToken;
@ -359,6 +387,10 @@ export class MatcherSelector<Type, StateToken = any> extends EventEmitter<EventM
}
}
sourceTrackers.forEach((tracker) => {
tracker.preserve = false;
})
// If in a sustain mode, no models for new sources may launch;
// only existing sequences are allowed to continue.
if(this.sustainMode && unmatchedSource) {
@ -653,7 +685,7 @@ export class MatcherSelector<Type, StateToken = any> extends EventEmitter<EventM
// If we just rejected the last possible matcher for a tracked gesture-source...
// then, for each such affected source...
for(const stat of remainingMatcherStats) {
if(stat.pendingCount == 0) {
if(stat.pendingCount == 0 && !stat.tracker.preserve) {
// ... report the failure and signal to close-out that source / stop tracking it.
stat.tracker.matchPromise.resolve({
matcher: null,

23
common/web/gesture-recognizer/src/engine/headless/gestures/specs/gestureModel.ts
View file

@ -142,7 +142,28 @@ export interface GestureModel<Type, StateToken = any> {
readonly resolutionAction: GestureResolutionSpec;
readonly rejectionActions?: Partial<Record<FulfillmentCause, RejectionReplace>>;
/*
Do NOT allow 'cancelled' rejection-actions. If 'cancelled', the corresponding `GestureSource`s
can no longer be valid matches for the GestureModel under any condition.
Generally, this is due to the underlying sources themselves being cancelled, but this can also
arise under the following combination of conditions:
- a model instantly rejects...
- whenever a new `GestureSource` starts and matches an instantly-rejecting `PathModel` for this
`GestureModel` (cause: 'path')
- when it fails initial-state validation (cause: 'item')
- a corresponding rejection action has been defined.
- For example, it also rejects under certain path conditions (for its original `GestureSource`)
that are recoverable.
Upon receiving an incoming extra GestureSource, the model would instantly reject (cause: 'path')
and could attempt to restart if specified to do so by a 'path' rejection action. In such a case,
it would instantly reject again due to the same reason. Instant rejection of a replacement model
during a rejection action is reported as 'cancellation'.
*/
readonly rejectionActions?: Partial<Record<Exclude<FulfillmentCause, 'cancelled'>, RejectionReplace>>;
// If there is a 'gesture stack' associated with the gesture chain, it's auto-popped
// upon completion of the chain. Optional-chaining can sustain the chain while the
// potential child gesture is still a possibility.

8
common/web/gesture-recognizer/src/engine/headless/inputEngineBase.ts
View file

@ -56,15 +56,17 @@ export abstract class InputEngineBase<HoveredItemType, StateToken = any> extends
return source;
}
public fulfillInputStart(touchpoint: GestureSource<HoveredItemType, StateToken>) {}
/**
* Calls to this method will cancel any touchpoints whose internal IDs are _not_ included in the parameter.
* Designed to facilitate recovery from error cases and peculiar states that sometimes arise when debugging.
* @param identifiers
*/
maintainTouchpointsWithIds(identifiers: number[]) {
const identifiersToMaintain = identifiers.map((internal_id) => this.identifierMap[internal_id]);
maintainTouchpoints(touchpoints: GestureSource<HoveredItemType, StateToken>[]) {
touchpoints ||= [];
this._activeTouchpoints
.filter((source) => !identifiersToMaintain.includes(source.rawIdentifier))
.filter((source) => !touchpoints.includes(source))
// Will trigger `.dropTouchpoint` later in the event chain.
.forEach((source) => source.terminate(true));
}

32
common/web/gesture-recognizer/src/engine/headless/touchpointCoordinator.ts
View file

@ -5,9 +5,9 @@ import { MatcherSelection, MatcherSelector } from "./gestures/matchers/matcherSe
import { GestureSequence } from "./gestures/matchers/gestureSequence.js";
import { GestureModelDefs, getGestureModel, getGestureModelSet } from "./gestures/specs/gestureModelDefs.js";
import { GestureModel } from "./gestures/specs/gestureModel.js";
import { timedPromise } from "@keymanapp/web-utils";
import { InputSample } from "./inputSample.js";
import { GestureDebugPath } from "./gestureDebugPath.js";
import { reportError } from "../reportError.js";
interface EventMap<HoveredItemType, StateToken> {
/**
@ -223,8 +223,35 @@ export class TouchpointCoordinator<HoveredItemType, StateToken=any> extends Even
const selector = this.currentSelector;
touchpoint.setGestureMatchInspector(this.buildGestureMatchInspector(selector));
this.emit('inputstart', touchpoint);
/*
If there's an error in code receiving this event, we must not let that break the flow of
event input processing - we may still have a locking Promise corresponding to our active
GestureSource. (See: next comment)
*/
try {
this.emit('inputstart', touchpoint);
} catch (err) {
reportError("Error from 'inputstart' event listener", err);
}
/*
If an `InputEventEngine` internally utilizes the `AsyncClosureDispatchQueue`, this is the point
at which we are now safe to process further events. The correct 'stateToken' has been identified
and all GestureMatcher possibilities for the source have been launched; path updates may resume _and_
new incoming paths may now be safely handled. As such, we can now fulfill any Promise returned by
a closure defined within its `inputStart` method for the `GestureSource` under consideration.
It is quite important that we _do_ fulfill the `Promise` if it exists - further event processing will
be blocked for such engines until this is done! (Hence the try-catch above)
*/
this.inputEngines.forEach((engine) => {
engine.fulfillInputStart(touchpoint);
});
// ----------------------------------------
// All gesture-matching is prepared; now we await the source's first gesture model match.
const selection = await selectionPromise;
// Any related 'push' mechanics that may still be lingering are currently handled by GestureSequence
@ -255,6 +282,7 @@ export class TouchpointCoordinator<HoveredItemType, StateToken=any> extends Even
// Could track sequences easily enough; the question is how to tell when to 'let go'.
// No try-catch because only there's no critical code after it.
this.emit('recognizedgesture', gestureSequence);
}

1
common/web/gesture-recognizer/src/engine/index.ts
View file

@ -1,5 +1,6 @@
import { validateModelDefs } from './headless/gestures/specs/modelDefValidator.js';
export { AsyncClosureDispatchQueue, QueueClosure } from './headless/asyncClosureDispatchQueue.js';
export { CumulativePathStats } from './headless/cumulativePathStats.js';
export { GestureModelDefs } from './headless/gestures/specs/gestureModelDefs.js';
export { GestureRecognizer } from "./gestureRecognizer.js";

75
common/web/gesture-recognizer/src/engine/inputEventEngine.ts
View file

@ -2,6 +2,7 @@ import { InputEngineBase } from "./headless/inputEngineBase.js";
import { InputSample } from "./headless/inputSample.js";
import { GestureSource } from "./headless/gestureSource.js";
import { GestureRecognizerConfiguration } from "./index.js";
import { reportError } from "./reportError.js";
export function processSampleClientCoords<Type, StateToken>(config: GestureRecognizerConfiguration<Type>, clientX: number, clientY: number) {
const targetRect = config.targetRoot.getBoundingClientRect();
@ -13,12 +14,12 @@ export function processSampleClientCoords<Type, StateToken>(config: GestureRecog
} as InputSample<Type, StateToken>;
}
export abstract class InputEventEngine<HoveredItemType, StateToken> extends InputEngineBase<HoveredItemType, StateToken> {
export abstract class InputEventEngine<ItemType, StateToken> extends InputEngineBase<ItemType, StateToken> {
abstract registerEventHandlers(): void;
abstract unregisterEventHandlers(): void;
protected buildSampleFor(clientX: number, clientY: number, target: EventTarget, timestamp: number, source: GestureSource<HoveredItemType, StateToken>): InputSample<HoveredItemType, StateToken> {
const sample: InputSample<HoveredItemType, StateToken> = {
protected buildSampleFor(clientX: number, clientY: number, target: EventTarget, timestamp: number, source: GestureSource<ItemType, StateToken>): InputSample<ItemType, StateToken> {
const sample: InputSample<ItemType, StateToken> = {
...processSampleClientCoords(this.config, clientX, clientY),
t: timestamp,
stateToken: source?.stateToken ?? this.stateToken
@ -31,7 +32,7 @@ export abstract class InputEventEngine<HoveredItemType, StateToken> extends Inpu
return sample;
}
protected onInputStart(identifier: number, sample: InputSample<HoveredItemType, StateToken>, target: EventTarget, isFromTouch: boolean) {
protected onInputStart(identifier: number, sample: InputSample<ItemType, StateToken>, target: EventTarget, isFromTouch: boolean) {
const touchpoint = this.createTouchpoint(identifier, isFromTouch);
touchpoint.update(sample);
@ -47,49 +48,49 @@ export abstract class InputEventEngine<HoveredItemType, StateToken> extends Inpu
this.dropTouchpoint(touchpoint);
});
this.emit('pointstart', touchpoint);
}
protected onInputMove(identifier: number, sample: InputSample<HoveredItemType, StateToken>, target: EventTarget) {
const activePoint = this.getTouchpointWithId(identifier);
if(!activePoint) {
return;
try {
this.emit('pointstart', touchpoint);
} catch(err) {
reportError('Engine-internal error while initializing gesture matching for new source', err);
}
activePoint.update(sample);
return touchpoint;
}
protected onInputMoveCancel(identifier: number, sample: InputSample<HoveredItemType, StateToken>, target: EventTarget) {
const touchpoint = this.getTouchpointWithId(identifier);
protected onInputMove(touchpoint: GestureSource<ItemType, StateToken>, sample: InputSample<ItemType, StateToken>, target: EventTarget) {
if(!touchpoint) {
return;
}
touchpoint.update(sample);
touchpoint.path.terminate(true);
}
protected onInputEnd(identifier: number, target: EventTarget) {
const touchpoint = this.getTouchpointWithId(identifier);
if(!touchpoint) {
return;
}
const lastEntry = touchpoint.path.stats.lastSample;
const sample = this.buildSampleFor(lastEntry.clientX, lastEntry.clientY, target, lastEntry.t, touchpoint);
/* While an 'end' event immediately follows a 'move' if it occurred simultaneously,
* this is decidedly _not_ the case if the touchpoint was held for a while without
* moving, even at the point of its release.
*
* We'll never need to worry about the touchpoint moving here, and thus we don't
* need to worry about `currentHoveredItem` changing. We're only concerned with
* recording the _timing_ of the touchpoint's release.
*/
if(sample.t != lastEntry.t) {
try {
touchpoint.update(sample);
} catch(err) {
reportError('Error occurred while updating source', err);
}
}
protected onInputMoveCancel(touchpoint: GestureSource<ItemType, StateToken>, sample: InputSample<ItemType, StateToken>, target: EventTarget) {
if(!touchpoint) {
return;
}
this.getTouchpointWithId(identifier)?.path.terminate(false);
try {
touchpoint.update(sample);
touchpoint.path.terminate(true);
} catch(err) {
reportError('Error occurred while cancelling further input for source', err);
}
}
protected onInputEnd(touchpoint: GestureSource<ItemType, StateToken>, target: EventTarget) {
if(!touchpoint) {
return;
}
try {
touchpoint.path.terminate(false);
} catch(err) {
reportError('Error occurred while finalizing input for source', err);
}
}
}

60
common/web/gesture-recognizer/src/engine/mouseEventEngine.ts
View file

@ -1,10 +1,11 @@
import { GestureRecognizerConfiguration } from "./configuration/gestureRecognizerConfiguration.js";
import { InputEventEngine } from "./inputEventEngine.js";
import { InputSample } from "./headless/inputSample.js";
import { Nonoptional } from "./nonoptional.js";
import { ZoneBoundaryChecker } from "./configuration/zoneBoundaryChecker.js";
import { GestureSource } from "./headless/gestureSource.js";
export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEventEngine<HoveredItemType, StateToken> {
// Does NOT use the AsyncClosureDispatchQueue... simply because there can only ever be one mouse touchpoint.
export class MouseEventEngine<ItemType, StateToken = any> extends InputEventEngine<ItemType, StateToken> {
private readonly _mouseStart: typeof MouseEventEngine.prototype.onMouseStart;
private readonly _mouseMove: typeof MouseEventEngine.prototype.onMouseMove;
private readonly _mouseEnd: typeof MouseEventEngine.prototype.onMouseEnd;
@ -12,7 +13,10 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
private hasActiveClick: boolean = false;
private disabledSafeBounds: number = 0;
public constructor(config: Nonoptional<GestureRecognizerConfiguration<HoveredItemType, StateToken>>) {
private currentSource: GestureSource<ItemType, StateToken> = null;
private readonly activeIdentifier = 0;
public constructor(config: Nonoptional<GestureRecognizerConfiguration<ItemType, StateToken>>) {
super(config);
// We use this approach, rather than .bind, because _this_ version allows hook
@ -25,20 +29,6 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
private get eventRoot(): HTMLElement {
return this.config.mouseEventRoot;
}
private get activeIdentifier(): number {
return 0;
}
// public static forPredictiveBanner(banner: SuggestionBanner, handlerRoot: SuggestionManager) {
// const config: GestureRecognizerConfiguration = {
// targetRoot: banner.getDiv(),
// // document.body is the event root b/c we need to track the mouse if it leaves
// // the VisualKeyboard's hierarchy.
// eventRoot: document.body,
// };
// return new MouseEventEngine(config);
// }
registerEventHandlers() {
this.eventRoot.addEventListener('mousedown', this._mouseStart, true);
@ -66,10 +56,9 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
}
}
private buildSampleFromEvent(event: MouseEvent, identifier: number) {
private buildSampleFromEvent(event: MouseEvent) {
// WILL be null for newly-starting `GestureSource`s / contact points.
const source = this.getTouchpointWithId(identifier);
return this.buildSampleFor(event.clientX, event.clientY, event.target, performance.now(), source);
return this.buildSampleFor(event.clientX, event.clientY, event.target, performance.now(), this.currentSource);
}
onMouseStart(event: MouseEvent) {
@ -81,8 +70,7 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
this.preventPropagation(event);
const identifier = this.activeIdentifier;
const sample = this.buildSampleFromEvent(event, identifier);
const sample = this.buildSampleFromEvent(event);
if(!ZoneBoundaryChecker.inputStartOutOfBoundsCheck(sample, this.config)) {
// If we started very close to a safe zone border, remember which one(s).
@ -90,36 +78,46 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
this.disabledSafeBounds = ZoneBoundaryChecker.inputStartSafeBoundProximityCheck(sample, this.config);
}
this.onInputStart(identifier, sample, event.target, false);
const touchpoint = this.onInputStart(this.activeIdentifier, sample, event.target, false);
this.currentSource = touchpoint;
const cleanup = () => {
this.currentSource = null;
}
touchpoint.path.on('complete', cleanup);
touchpoint.path.on('invalidated', cleanup);
}
onMouseMove(event: MouseEvent) {
if(!this.hasActiveTouchpoint(this.activeIdentifier)) {
const source = this.currentSource;
if(!source) {
return;
}
const sample = this.buildSampleFromEvent(event, this.activeIdentifier);
const sample = this.buildSampleFromEvent(event);
if(!event.buttons) {
if(this.hasActiveClick) {
this.hasActiveClick = false;
this.onInputMoveCancel(this.activeIdentifier, sample, event.target);
this.onInputMoveCancel(source, sample, event.target);
}
return;
}
this.preventPropagation(event);
const config = this.getConfigForId(this.activeIdentifier);
const config = source.currentRecognizerConfig;
if(!ZoneBoundaryChecker.inputMoveCancellationCheck(sample, config, this.disabledSafeBounds)) {
this.onInputMove(this.activeIdentifier, sample, event.target);
this.onInputMove(source, sample, event.target);
} else {
this.onInputMoveCancel(this.activeIdentifier, sample, event.target);
this.onInputMoveCancel(source, sample, event.target);
}
}
onMouseEnd(event: MouseEvent) {
if(!this.hasActiveTouchpoint(this.activeIdentifier)) {
const source = this.currentSource;
if(!source) {
return;
}
@ -127,6 +125,6 @@ export class MouseEventEngine<HoveredItemType, StateToken = any> extends InputEv
this.hasActiveClick = false;
}
this.onInputEnd(this.activeIdentifier, event.target);
this.onInputEnd(source, event.target);
}
}

9
common/web/gesture-recognizer/src/engine/reportError.ts Normal file
View file

@ -0,0 +1,9 @@
export function reportError(baseMsg: string, err: Error) {
// Our mobile-app Sentry logging will listen for this and log it.
if(err instanceof Error) {
console.error(`${baseMsg}: ${err.message}\n\n${err.stack}`);
} else {
console.error(baseMsg);
console.error(err);
}
}

325
common/web/gesture-recognizer/src/engine/touchEventEngine.ts
View file

@ -1,9 +1,11 @@
import { GestureRecognizerConfiguration } from "./configuration/gestureRecognizerConfiguration.js";
import { InputEventEngine } from "./inputEventEngine.js";
import { InputSample } from "./headless/inputSample.js";
import { Nonoptional } from "./nonoptional.js";
import { ZoneBoundaryChecker } from "./configuration/zoneBoundaryChecker.js";
import { GestureSource } from "./headless/gestureSource.js";
import { ManagedPromise } from "@keymanapp/web-utils";
import { AsyncClosureDispatchQueue } from "./headless/asyncClosureDispatchQueue.js";
import { GesturePath } from "./index.js";
function touchListToArray(list: TouchList) {
const arr: Touch[] = [];
@ -14,14 +16,19 @@ function touchListToArray(list: TouchList) {
return arr;
}
export class TouchEventEngine<HoveredItemType, StateToken = any> extends InputEventEngine<HoveredItemType, StateToken> {
export class TouchEventEngine<ItemType, StateToken = any> extends InputEventEngine<ItemType, StateToken> {
private readonly _touchStart: typeof TouchEventEngine.prototype.onTouchStart;
private readonly _touchMove: typeof TouchEventEngine.prototype.onTouchMove;
private readonly _touchEnd: typeof TouchEventEngine.prototype.onTouchEnd;
private safeBoundMaskMap: {[id: number]: number} = {};
protected readonly eventDispatcher = new AsyncClosureDispatchQueue();
public constructor(config: Nonoptional<GestureRecognizerConfiguration<HoveredItemType, StateToken>>) {
private safeBoundMaskMap: {[id: number]: number} = {};
// This map works synchronously with the actual event handlers.
private pendingSourcePromises: Map<number, ManagedPromise<GestureSource<ItemType, StateToken>>> = new Map();
private inputStartSignalMap: Map<GestureSource<ItemType, StateToken>, ManagedPromise<void>> = new Map();
public constructor(config: Nonoptional<GestureRecognizerConfiguration<ItemType, StateToken>>) {
super(config);
// We use this approach, rather than .bind, because _this_ version allows hook
@ -35,17 +42,6 @@ export class TouchEventEngine<HoveredItemType, StateToken = any> extends InputEv
return this.config.touchEventRoot;
}
// public static forPredictiveBanner(banner: SuggestionBanner, handlerRoot: SuggestionManager) {
// const config: GestureRecognizerConfiguration = {
// targetRoot: banner.getDiv(),
// // document.body is the event root b/c we need to track the mouse if it leaves
// // the VisualKeyboard's hierarchy.
// eventRoot: banner.getDiv(),
// };
// return new TouchEventEngine(config);
// }
registerEventHandlers() {
// The 'passive' property ensures we can prevent MouseEvent followups from TouchEvents.
// It is only specified during `addEventListener`, not during `removeEventListener`.
@ -76,7 +72,7 @@ export class TouchEventEngine<HoveredItemType, StateToken = any> extends InputEv
}
}
public dropTouchpoint(source: GestureSource<HoveredItemType>) {
public dropTouchpoint(source: GestureSource<ItemType>) {
super.dropTouchpoint(source);
for(const key of Object.keys(this.safeBoundMaskMap)) {
@ -86,9 +82,21 @@ export class TouchEventEngine<HoveredItemType, StateToken = any> extends InputEv
}
}
private buildSampleFromTouch(touch: Touch, timestamp: number) {
public fulfillInputStart(touchpoint: GestureSource<ItemType, StateToken, GesturePath<ItemType, StateToken>>) {
const lock = this.inputStartSignalMap.get(touchpoint);
if(lock) {
this.inputStartSignalMap.delete(touchpoint);
lock.resolve();
}
};
public hasActiveTouchpoint(identifier: number): boolean {
const baseResult = super.hasActiveTouchpoint(identifier);
return baseResult || !!this.pendingSourcePromises.has(identifier);
}
private buildSampleFromTouch(touch: Touch, timestamp: number, source: GestureSource<ItemType, StateToken>) {
// WILL be null for newly-starting `GestureSource`s / contact points.
const source = this.getTouchpointWithId(touch.identifier);
return this.buildSampleFor(touch.clientX, touch.clientY, touch.target, timestamp, source);
}
@ -106,89 +114,244 @@ export class TouchEventEngine<HoveredItemType, StateToken = any> extends InputEv
// during a touchstart.)
const allTouches = touchListToArray(event.touches);
const newTouches = touchListToArray(event.changedTouches);
// Maintain all touches in the `.touches` array that are NOT marked as `.changedTouches` (and therefore, new)
this.maintainTouchpointsWithIds(allTouches
.filter((touch1) => newTouches.findIndex(touch2 => touch1.identifier == touch2.identifier) == -1)
.map((touch) => touch.identifier)
);
const oldTouches = allTouches.filter((touch1) => {
return newTouches.findIndex(touch2 => touch1.identifier == touch2.identifier) == -1;
});
// Ensure the same timestamp is used for all touches being updated.
const timestamp = performance.now();
// Any 'old touches' should have pre-existing entries in our promise-map that are still current, as
// the promise-map is maintained 100% synchronously with incoming events.
const oldSourcePromises = oldTouches.map((touch) => this.pendingSourcePromises.get(touch.identifier));
// During a touch-start, only _new_ touch contact points are listed here;
// we shouldn't signal "input start" for any previously-existing touch points,
// so `.changedTouches` is the best way forward.
this.eventDispatcher.runAsync(async () => {
const oldSources = await Promise.all(oldSourcePromises);
// Maintain all touches in the `.touches` array that are NOT marked as `.changedTouches` (and therefore, new)
this.maintainTouchpoints(oldSources);
return this.eventDispatcher.defaultWait;
});
/*
We create Promises that can be set and retrieved synchronously with the actual event handlers
in order to prevent issues from tricky asynchronous identifier-to-source mapping attempts.
As these Promises are set (and thus, retrievable) synchronously with the actual event handlers,
we can closure-capture them for use in the internally-asynchronous processing closures.
`capturedSourcePromises` will be useful for closure-capture binding the new Promise(s) to
the closure to be queued. `this.pendingSourcePromises` facilitates similar closure-capture
patterns within the touchMove and touchEnd handlers for their queued closures.
*/
const capturedSourcePromises = new Map<number, ManagedPromise<GestureSource<ItemType, StateToken>>>();
for(let i=0; i < event.changedTouches.length; i++) {
const touch = event.changedTouches.item(i);
const sample = this.buildSampleFromTouch(touch, timestamp);
const promise = new ManagedPromise<GestureSource<ItemType, StateToken>>();
this.pendingSourcePromises.set(touch.identifier, promise);
capturedSourcePromises.set(touch.identifier, promise);
}
if(!ZoneBoundaryChecker.inputStartOutOfBoundsCheck(sample, this.config)) {
// If we started very close to a safe zone border, remember which one(s).
// This is important for input-sequence cancellation check logic.
this.safeBoundMaskMap[touch.identifier] = ZoneBoundaryChecker.inputStartSafeBoundProximityCheck(sample, this.config);
} else {
// This touchpoint shouldn't be considered; do not signal a touchstart for it.
continue;
/*
When multiple touchpoints are active, we need to ensure a specific order of events.
The easiest way to ensure the exact order involves programmatic delay of their
processing, essentially "sequentializing" the events into a deterministic order.
It also helps to ensure that any path updates are only emitted when all listeners
for that path have been prepared - and other parts of the engine cause that to happen
asynchronously in certain situations. Within KMW, one such case is when a simple-tap
with `nextLayer` defined is auto-completed by a new incoming touch, triggering an
instant layer-change.
*/
this.eventDispatcher.runAsync(() => {
// Ensure the same timestamp is used for all touches being updated.
const timestamp = performance.now();
let touchpoint: GestureSource<ItemType, StateToken> = null;
// During a touch-start, only _new_ touch contact points are listed here;
// we shouldn't signal "input start" for any previously-existing touch points,
// so `.changedTouches` is the best way forward.
for(let i=0; i < event.changedTouches.length; i++) {
const touch = event.changedTouches.item(i);
const touchId = touch.identifier;
const sample = this.buildSampleFromTouch(touch, timestamp, null);
if(!ZoneBoundaryChecker.inputStartOutOfBoundsCheck(sample, this.config)) {
// If we started very close to a safe zone border, remember which one(s).
// This is important for input-sequence cancellation check logic.
this.safeBoundMaskMap[touchId] = ZoneBoundaryChecker.inputStartSafeBoundProximityCheck(sample, this.config);
} else {
// This touchpoint shouldn't be considered; do not signal a touchstart for it.
let sourcePromise = capturedSourcePromises.get(touchId);
sourcePromise.resolve(null);
continue;
}
touchpoint = this.onInputStart(touchId, sample, event.target, true);
/*
We use the closure-captured version bound to this specific closure, rather than the
most recent one for the touch-identifier - under heavy rapid typing, it's possible that
the touch-identifier has been reused.
The resolved Promise may then be used to retrieve the correct source in the other event
handlers' closures.
*/
let sourcePromise = capturedSourcePromises.get(touchId);
sourcePromise.resolve(touchpoint);
/*
Ensure we only do the cleanup if and when it hasn't already been replaced by new events later.
Must be done for EACH source - we can't risk leaving a lingering entry once we've dismissed
processing for the source. Failure to do so may result in blocking touch events that should
no longer be manipulated by this engine by affecting `hasActiveTouchpoint`.
*/
const cleanup = () => {
/*
If delays accumulate significantly, it is possible that when this queued closure is run,
a different touchpoint is reusing the same identifier. Don't delete the entry if our
entry has been replaced.
*/
if(this.pendingSourcePromises.get(touchId) == sourcePromise) {
this.pendingSourcePromises.delete(touchId);
}
}
touchpoint.path.on('complete', cleanup);
touchpoint.path.on('invalidated', cleanup);
}
this.onInputStart(touch.identifier, sample, event.target, true);
}
if(touchpoint) {
// This 'lock' should only be released when the last simultaneously-registered touch is published via
// gesture-recognizer event.
let eventSignalPromise = new ManagedPromise<void>();
this.inputStartSignalMap.set(touchpoint, eventSignalPromise);
return eventSignalPromise.corePromise;
}
});
}
onTouchMove(event: TouchEvent) {
let propagationActive = true;
// Ensure the same timestamp is used for all touches being updated.
const timestamp = performance.now();
this.maintainTouchpointsWithIds(touchListToArray(event.touches)
.map((touch) => touch.identifier)
);
// Do not change to `changedTouches` - we need a sample for all active touches in order
// to facilitate path-update synchronization for multi-touch gestures.
//
// May be worth doing changedTouches _first_ though.
for(let i=0; i < event.touches.length; i++) {
for(let i = 0; i < event.touches.length; i++) {
const touch = event.touches.item(i);
if(!this.hasActiveTouchpoint(touch.identifier)) {
continue;
}
if(propagationActive) {
if(this.hasActiveTouchpoint(touch.identifier)) {
this.preventPropagation(event);
propagationActive = false;
}
const config = this.getConfigForId(touch.identifier);
const sample = this.buildSampleFromTouch(touch, timestamp);
if(!ZoneBoundaryChecker.inputMoveCancellationCheck(sample, config, this.safeBoundMaskMap[touch.identifier])) {
this.onInputMove(touch.identifier, sample, touch.target);
} else {
this.onInputMoveCancel(touch.identifier, sample, touch.target);
break;
}
}
/*
Using the Promise map built in touchStart, we can retrieve a Promise for the source linked
to this event and closure-capture it for the closure queued below.
*/
const capturedSourcePromises = new Map<number, Promise<GestureSource<ItemType, StateToken>>>();
for(let i = 0; i < event.touches.length; i++) {
const touchId = event.touches.item(i).identifier;
// If the source's gesture is finalized or cancelled but touch events are ongoing,
// with no delay between event and its processing, the map entry here will be cleared.
capturedSourcePromises.set(touchId, this.pendingSourcePromises.get(touchId)?.corePromise);
}
this.eventDispatcher.runAsync(async () => {
const touches = await Promise.all(capturedSourcePromises.values());
this.maintainTouchpoints(touches);
return this.eventDispatcher.defaultWait;
});
/*
When multiple touchpoints are active, we need to ensure a specific order of events.
The easiest way to ensure the exact order involves programmatic delay of their
processing, essentially "sequentializing" the events into a deterministic order.
It also helps to ensure that any path updates are only emitted when all listeners
for that path have been prepared - and other parts of the engine cause that to happen
asynchronously in certain situations. Within KMW, one such case is when a simple-tap
with `nextLayer` defined is auto-completed by a new incoming touch, triggering an
instant layer-change.
*/
this.eventDispatcher.runAsync(async () => {
// Ensure the same timestamp is used for all touches being updated.
const timestamp = performance.now();
// Do not change to `changedTouches` - we need a sample for all active touches in order
// to facilitate path-update synchronization for multi-touch gestures.
//
// May be worth doing changedTouches _first_ though.
for(let i=0; i < event.touches.length; i++) {
const touch = event.touches.item(i);
const touchId = touch.identifier;
// Only lists touch contact points that have been lifted; touchmove is
// raised separately if any movement occurred.
//
// If the promise object could not be assigned, we `await undefined` -
// which JS converts to `await Promise.resolve(undefined)`. It's safe.
const source = await capturedSourcePromises.get(touchId);
if(!source || source.isPathComplete) {
continue;
}
const config = source.currentRecognizerConfig;
const sample = this.buildSampleFromTouch(touch, timestamp, source);
if(!ZoneBoundaryChecker.inputMoveCancellationCheck(sample, config, this.safeBoundMaskMap[touchId])) {
this.onInputMove(source, sample, touch.target);
} else {
this.onInputMoveCancel(source, sample, touch.target);
}
}
/*
Since we're operating within an async function, a Promise return-type
is implied. That cancels out the default wait, but we want to ensure
that the default wait is applied here.
*/
return this.eventDispatcher.defaultWait;
});
}
onTouchEnd(event: TouchEvent) {
let propagationActive = true;
// Only lists touch contact points that have been lifted; touchmove is raised separately if any movement occurred.
for(let i=0; i < event.changedTouches.length; i++) {
for(let i = 0; i < event.changedTouches.length; i++) {
const touch = event.changedTouches.item(i);
if(!this.hasActiveTouchpoint(touch.identifier)) {
continue;
}
if(propagationActive) {
if(this.hasActiveTouchpoint(touch.identifier)) {
this.preventPropagation(event);
propagationActive = false;
break;
}
}
/*
Using the Promise map built in touchStart, we can retrieve a Promise for the source linked
to this event and closure-capture it for the closure queued below.
*/
const capturedSourcePromises = new Map<number, Promise<GestureSource<ItemType, StateToken>>>();
// Any ending touches don't show up in event.touches - only in event.changedTouches!
for(let i = 0; i < event.changedTouches.length; i++) {
const touchId = event.changedTouches.item(i).identifier;
// If the source's gesture is finalized or cancelled but touch events are ongoing,
// with no delay between event and its processing, the map entry here will be cleared.
const promiseToCapture = this.pendingSourcePromises.get(touchId)?.corePromise;
capturedSourcePromises.set(touchId, promiseToCapture);
}
this.eventDispatcher.runAsync(async () => {
// Only lists touch contact points that have been lifted; touchmove is
// raised separately if any movement occurred.
//
// If the promise object could not be assigned, we `await undefined` -
// which JS converts to `await Promise.resolve(undefined)`. It's safe.
for(let i=0; i < event.changedTouches.length; i++) {
const touch = event.changedTouches.item(i);
const source = await capturedSourcePromises.get(touch.identifier);
if(!source || source.isPathComplete) {
continue;
}
this.onInputEnd(source, event.target);
}
this.onInputEnd(touch.identifier, event.target);
}
return this.eventDispatcher.defaultWait;
});
}
}

14
common/web/gesture-recognizer/src/test/auto/browser/cases/canary.js
View file

@ -78,10 +78,14 @@ describe("'Canary' checks", function() {
// Ensure that the expected handler is called.
let fakeHandler = sinon.fake();
this.controller.recognizer.on('inputstart', fakeHandler)
this.controller.recognizer.on('inputstart', fakeHandler);
fireEvent();
await Promise.resolve();
await new Promise((resolve) => {
window.setTimeout(resolve, 0);
}).then(() => new Promise((resolve) => {
window.setTimeout(resolve, 0);
}));
assert.isTrue(fakeHandler.called, "Unit test attempt failed: handler was not called successfully.");
});
@ -99,10 +103,12 @@ describe("'Canary' checks", function() {
// Ensure that the expected handler is called.
let fakeHandler = sinon.fake();
this.controller.recognizer.on('inputstart', fakeHandler)
this.controller.recognizer.on('inputstart', fakeHandler);
fireEvent();
await Promise.resolve();
await new Promise((resolve) => {
window.setTimeout(resolve, 0);
});
assert.isTrue(fakeHandler.called, "Unit test attempt failed: handler was not called successfully.");
});

8
common/web/gesture-recognizer/src/test/auto/browser/cases/recordedCoordSequences.js
View file

@ -91,12 +91,12 @@ describe("Layer one - DOM -> InputSequence", function() {
const sampleResult = resultContactPath[j];
const sampleOriginal = originalContactPath[j];
assert.isOk(sampleResult, `An expected sample was missing during simulation - failed at path entry ${j}`);
assert.isOk(sampleOriginal, `An extra sample was generated during simulation - failed at path entry ${j}`);
assert.isOk(sampleResult, `An expected sample was missing during simulation - failed at path entry ${j}, path ${i}`);
assert.isOk(sampleOriginal, `An extra sample was generated during simulation - failed at path entry ${j}, path ${i}`);
// During test runs against a real Android device, we tend to get almost, but not-quite, integer targetX and targetY values.
expect(sampleResult.targetX).to.be.closeTo(sampleOriginal.targetX, 1e-4, `Mismatch in x-coord at path entry ${j}`);
expect(sampleResult.targetY).to.be.closeTo(sampleOriginal.targetY, 1e-4, `Mismatch in y-coord at path entry ${j}`);
expect(sampleResult.targetX).to.be.closeTo(sampleOriginal.targetX, 1e-4, `Mismatch in x-coord at path entry ${j}, path ${i}`);
expect(sampleResult.targetY).to.be.closeTo(sampleOriginal.targetY, 1e-4, `Mismatch in y-coord at path entry ${j}, path ${i}`);
}
}

236
common/web/gesture-recognizer/src/test/auto/headless/asyncClosureDispatchQueue.spec.ts Normal file
View file

@ -0,0 +1,236 @@
import { assert } from 'chai'
import { default as sinon, type SinonSpy } from 'sinon';
import { AsyncClosureDispatchQueue, type QueueClosure } from '@keymanapp/gesture-recognizer';
import { ManagedPromise, timedPromise } from '@keymanapp/web-utils';
type ClosureSpy = SinonSpy<[], ReturnType<QueueClosure>>;
describe('AsyncClosureDispatchQueue', () => {
it('has expected initial state', () => {
const queue = new AsyncClosureDispatchQueue();
assert.isTrue(queue.ready);
});
it('proper handling of simple closure when queue is empty (default config)', async () => {
const queue = new AsyncClosureDispatchQueue();
const fake = sinon.fake();
queue.runAsync(() => {
fake(queue.ready);
});
assert.isFalse(queue.ready);
// If the queue was ready, this should be called immediately.
assert.isTrue(fake.called);
// Default delay between entries: a macroqueue task (i.e., setTimeout(0))
await timedPromise(0);
assert.isTrue(queue.ready);
assert.isTrue(fake.called);
// During the actual closure call, the queue is still awaiting async completion of the closure.
// Default wait: a macroqueue task
assert.isFalse(fake.firstCall.args[0]);
});
it('proper handling of Promise-returning closure when queue is empty (default config)', async () => {
const queue = new AsyncClosureDispatchQueue();
const lock = new ManagedPromise<void>();
const fake = sinon.fake();
queue.runAsync(() => {
fake(queue.ready);
return lock.corePromise;
});
assert.isFalse(queue.ready);
// If the queue was ready, this should be called immediately.
assert.isTrue(fake.called);
assert.isFalse(fake.firstCall.args[0]);
await timedPromise(50);
assert.isFalse(queue.ready);
lock.resolve();
// Allow the newly-resolved Promise to chain.
// ("White-box" info here, but once is enough.)
await Promise.resolve();
assert.isTrue(queue.ready);
assert.isTrue(fake.called);
});
it('proper handling of simple closure when queue is not empty (default config)', async () => {
const queue = new AsyncClosureDispatchQueue();
const fakeTimers = sinon.useFakeTimers();
const lock = new ManagedPromise<void>();
queue.runAsync(() => lock.corePromise);
const fake = sinon.fake();
queue.runAsync(() => {
fake(queue.ready);
});
try {
assert.isFalse(queue.ready);
assert.isFalse(fake.called);
// Doesn't matter how long we wait; there's still a pending entry in front of `fake`.
await fakeTimers.tickAsync(50);
assert.isFalse(queue.ready);
assert.isFalse(fake.called);
// Allow that pending entry to resolve; `fake` should be able to resolve afterward with little issue.
lock.resolve();
await fakeTimers.tickAsync(50);
assert.isTrue(queue.ready);
assert.isTrue(fake.called);
// During the actual closure call, the queue is still awaiting async completion of the closure.
// Default wait: a macroqueue task
assert.isFalse(fake.firstCall.args[0]);
} finally {
fakeTimers.restore();
}
});
it('complex case 1 - many tasks, all queued at the same time', async () => {
// Uses the default timeout between events; just making it extra-explicit here.
const queue = new AsyncClosureDispatchQueue(() => { return timedPromise(0) });
const buildSet = (n: number) => {
let set: ClosureSpy[] = [];
// Deliberately using the same one multiple times - the class gives us a call count.
let closure = sinon.spy(() => {});
for(let i=0; i < n; i++) {
set.push(closure);
}
return set;
}
const set0 = buildSet(3);
const lock0 = new ManagedPromise<void>();
const set1 = buildSet(7);
const lock1 = new ManagedPromise<void>();
const set2 = buildSet(5);
const fakeTimers = sinon.useFakeTimers();
set0.forEach((entry) => queue.runAsync(entry));
queue.runAsync(() => {
return lock0.corePromise;
});
set1.forEach((entry) => queue.runAsync(entry));
queue.runAsync(() => {
return lock1.corePromise;
})
set2.forEach((entry) => queue.runAsync(entry));
assert.isFalse(queue.ready);
try {
// Run set0; it'll stop before set1 due to lock0 not being resolved.
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 0);
assert.equal(set2[0].callCount, 0);
// Now we run set1; it'll stop before set2 due to lock1 not being resolved.
lock0.resolve();
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 7);
assert.equal(set2[0].callCount, 0);
// Now we run set2, flushing out the queue.
lock1.resolve();
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 7);
assert.equal(set2[0].callCount, 5);
assert.isTrue(queue.ready);
} finally {
fakeTimers.restore();
}
});
it('complex case 2 - queued closure promises "unlocking" out of order', async () => {
// Uses the default timeout between events; just making it extra-explicit here.
const queue = new AsyncClosureDispatchQueue(() => { return timedPromise(0) });
const buildSet = (n: number) => {
let set: ClosureSpy[] = [];
// Deliberately using the same one multiple times - the class gives us a call count.
let closure = sinon.spy(() => {});
for(let i=0; i < n; i++) {
set.push(closure);
}
return set;
}
const fakeTimers = sinon.useFakeTimers();
const set0 = buildSet(3);
const lock0 = new ManagedPromise<void>();
const set1 = buildSet(7);
const lock1 = new ManagedPromise<void>();
const set2 = buildSet(5);
set0.forEach((entry) => queue.runAsync(entry));
queue.runAsync(() => {
return lock0.corePromise;
});
set1.forEach((entry) => queue.runAsync(entry));
queue.runAsync(() => {
return lock1.corePromise;
})
set2.forEach((entry) => queue.runAsync(entry));
assert.isFalse(queue.ready);
try {
// Run set0; it'll stop before set1 due to lock0 not being resolved.
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 0);
assert.equal(set2[0].callCount, 0);
// Now we resolve lock1 - but this isn't what is currently blocking the queue.
// No new tasks should run.
lock1.resolve(); /* NOTE: is being unlocked before lock0, which is earlier! */
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 0);
assert.equal(set2[0].callCount, 0);
// Now we resolve lock0, allowing both set1 and set2 to complete.
lock0.resolve();
await fakeTimers.tickAsync(50);
assert.equal(set0[0].callCount, 3);
assert.equal(set1[0].callCount, 7);
assert.equal(set2[0].callCount, 5);
assert.isTrue(queue.ready);
} finally {
fakeTimers.restore();
}
});
});

2
common/web/gesture-recognizer/src/tools/unit-test-resources/src/headlessInputEngine.ts
View file

@ -85,7 +85,7 @@ export class HeadlessInputEngine<Type = any> extends InputEngineBase<Type> {
const playbackTerminations = recordedObj.inputs.map((recording, index) => this.playbackTerminations(sources[index], recording));
playbackStarts.forEach((promise) => promise.then(() => {
this.maintainTouchpointsWithIds(playbackStartTuples.map((tuple) => tuple.internal_id));
this.maintainTouchpoints(playbackStartTuples.map((tuple) => tuple.source));
})
);

3
common/web/gesture-recognizer/src/tools/unit-test-resources/src/inputSequenceSimulator.ts
View file

@ -111,7 +111,8 @@ export class InputSequenceSimulator<HoveredItemType> {
let touchEventDict: TouchEventInit = {
bubbles: true,
touches: changedTouches.concat(otherTouches),
// Ending touchpoints should NOT show up in `touches`.
touches: state == 'end' ? otherTouches : changedTouches.concat(otherTouches),
changedTouches: changedTouches,
}

10
common/web/keyboard-processor/src/text/codes.ts
View file

@ -1,7 +1,7 @@
// TODO: Move to separate folder: 'codes'
// We should start splitting off code needed by keyboards even without a KeyboardProcessor active.
// There's an upcoming `/common/web/types` package that 'codes' and 'keyboards' may fit well within.
// In fact, there's a file there (on its branch) that should be merged with this one!
// see also: common/web/types/src/kmx/kmx.ts
const Codes = {
// Define Keyman Developer modifier bit-flags (exposed for use by other modules)
@ -25,6 +25,10 @@ const Codes = {
"NO_SCROLL_LOCK":0x2000, // NOTSCROLLFLAG
"VIRTUAL_KEY":0x4000, // ISVIRTUALKEY
"VIRTUAL_CHAR_KEY":0x8000 // VIRTUALCHARKEY // Unused by KMW, but reserved for use by other Keyman engines.
// Note: keys_mod_other = 0x10000, used by KMX+ for the
// other modifier flag in layers, > 16 bit so not available here.
// See keys_mod_other in keyman_core_ldml.ts
},
modifierBitmasks: {
@ -168,4 +172,4 @@ const Codes = {
}
}
export default Codes;
export default Codes;

14
common/web/keyboard-processor/src/text/outputTarget.ts
View file

@ -13,7 +13,7 @@ import { Deadkey, DeadkeyTracker } from "./deadkeys.js";
export function isEmptyTransform(transform: Transform) {
if(!transform) {
return false;
return true;
}
return transform.insert === '' && transform.deleteLeft === 0 && (transform.deleteRight ?? 0) === 0;
}
@ -21,15 +21,17 @@ export function isEmptyTransform(transform: Transform) {
export class TextTransform implements Transform {
readonly insert: string;
readonly deleteLeft: number;
readonly deleteRight?: number;
readonly deleteRight: number;
readonly erasedSelection: boolean;
constructor(insert: string, deleteLeft: number, deleteRight?: number) {
constructor(insert: string, deleteLeft: number, deleteRight: number, erasedSelection: boolean) {
this.insert = insert;
this.deleteLeft = deleteLeft;
this.deleteRight = deleteRight || 0;
this.deleteRight = deleteRight;
this.erasedSelection = erasedSelection;
}
public static readonly nil = new TextTransform('', 0, 0);
public static readonly nil = new TextTransform('', 0, 0, false);
}
export class Transcription {
@ -138,7 +140,7 @@ export default abstract class OutputTarget {
// caret mid-word..
const deletedRight = fromRight.substring(0, rightDivergenceIndex + 1)._kmwLength();
return new TextTransform(insertedText, deletedLeft, deletedRight);
return new TextTransform(insertedText, deletedLeft, deletedRight, original.getSelectedText() && !this.getSelectedText());
}
buildTranscriptionFrom(original: OutputTarget, keyEvent: KeyEvent, readonly: boolean, alternates?: Alternate[]): Transcription {

6
common/web/keyboard-processor/tests/node/transcriptions.js
View file

@ -448,7 +448,8 @@ but not himself.`; // Sheev Palpatine, in the Star Wars prequels.
assert.deepEqual(transform, {
insert: '',
deleteLeft: 0,
deleteRight: 0
deleteRight: 0,
erasedSelection: true
});
});
@ -459,7 +460,8 @@ but not himself.`; // Sheev Palpatine, in the Star Wars prequels.
const transform = {
insert: '',
deleteLeft: 0,
deleteRight: 0
deleteRight: 0,
erasedSelection: true
};
target.apply(transform);

35
common/web/lm-worker/src/main/correction/context-tracker.ts
View file

@ -406,6 +406,22 @@ export class ContextTracker extends CircularArray<TrackedContextState> {
* - For languages using whitespace to word-break, said keystroke would have to include said whitespace to break the assumption.
*/
function maintainLastToken() {
if(isWhitespace && editPath[tailIndex] == 'match') {
/*
We can land here if there are multiple whitespaces in a row.
There's already an implied whitespace to the left, so we conceptually
merge the new whitespace with that one.
*/
return state;
} else if(isBackspace) {
// Consider backspace entry for this case?
state.replaceTailForBackspace(finalToken, primaryInput.id);
} else {
state.updateTail(primaryInput ? transformDistribution : null, finalToken);
}
}
// If there is/was more than one context token available...
if(editPath.length > 1) {
// We're removing a context token, but at least one remains.
@ -438,14 +454,11 @@ export class ContextTracker extends CircularArray<TrackedContextState> {
}
state.pushTail(pushedToken);
} else { // We're editing the final context token.
} else {
// We're editing the final context token.
// TODO: Assumption: we didn't 'miss' any inputs somehow.
// As is, may be prone to fragility should the lm-layer's tracked context 'desync' from its host's.
if(isBackspace) {
state.replaceTailForBackspace(finalToken, primaryInput.id);
} else {
state.updateTail(primaryInput ? transformDistribution : null, finalToken);
}
maintainLastToken();
}
// There is only one word in the context.
} else {
@ -458,13 +471,9 @@ export class ContextTracker extends CircularArray<TrackedContextState> {
token.raw = tokenizedContext[0];
token.transformDistributions = [transformDistribution];
state.pushTail(token);
} else { // Edit the lone context token.
// Consider backspace entry for this case?
if(isBackspace) {
state.replaceTailForBackspace(finalToken, primaryInput.id);
} else {
state.updateTail(primaryInput ? transformDistribution : null, finalToken);
}
} else {
// Edit the lone context token.
maintainLastToken();
}
}
return state;

5
common/web/types/src/kmx/element-string.ts
View file

@ -74,6 +74,11 @@ export class ElementString extends Array<ElemElement> {
typeFlag |= constants.elem_flags_type_uset;
// TODO-LDML: err on max buffer size
const needRanges = sections.usetparser.sizeUnicodeSet(item.segment);
if (needRanges < 0) {
// Note that sizeUnicodeSet() already will notify via callback if there's an
// error. So we can just exit here.
return null; // UnicodeSet error
}
const uset = sections.usetparser.parseUnicodeSet(item.segment, needRanges);
if (!uset) {
return null; // UnicodeSet error already thrown

16
common/web/types/src/kmx/kmx-plus.ts
View file

@ -2,7 +2,7 @@ import { constants } from '@keymanapp/ldml-keyboard-constants';
import * as r from 'restructure';
import { ElementString } from './element-string.js';
import { ListItem } from './string-list.js';
import { isOneChar, toOneChar, unescapeString } from '../util/util.js';
import { isOneChar, toOneChar, unescapeString, escapeStringForRegex } from '../util/util.js';
import { KMXFile } from './kmx.js';
import { UnicodeSetParser, UnicodeSet } from '@keymanapp/common-types';
import { VariableParser } from '../ldml-keyboard/pattern-parser.js';
@ -272,7 +272,7 @@ export class Vars extends Section {
return v.value.value; // string value
});
}
substituteStrings(str: string, sections: DependencySections): string {
substituteStrings(str: string, sections: DependencySections, forMatch?: boolean): string {
if (!str) return str;
return str.replaceAll(VariableParser.STRING_REFERENCE, (_entire, id) => {
const val = this.findStringVariableValue(id);
@ -280,6 +280,7 @@ export class Vars extends Section {
// Should have been caught during validation.
throw Error(`Internal Error: reference to missing string variable ${id}`);
}
if (forMatch) return escapeStringForRegex(val);
return val;
});
}
@ -292,8 +293,9 @@ export class Vars extends Section {
const set = Vars.findVariable(this.sets, id);
if (set !== null) {
const { items } = set;
const inner = items.map(i => i.value.value).join('|');
return `(?:${inner})`; // TODO-LDML: need to escape here
const escapedStrings = items.map(v => escapeStringForRegex(v.value.value));
const inner = escapedStrings.join('|');
return `(?:${inner})`;
}
// try as unicodeset
@ -372,11 +374,15 @@ export class UnicodeSetItem extends VarsItem {
};
export class SetVarItem extends VarsItem {
constructor(id: string, value: string[], sections: DependencySections) {
constructor(id: string, value: string[], sections: DependencySections, rawItems: string[]) {
super(id, value.join(' '), sections);
this.items = sections.elem.allocElementString(sections, value);
this.rawItems = rawItems;
}
// element string array
items: ElementString;
// like items, but with unprocessed marker strings
rawItems: string[];
valid() : boolean {
return !!this.items;
}

6
common/web/types/src/kmx/kmx.ts
View file

@ -345,6 +345,10 @@ export class KMXFile {
public static readonly ISVIRTUALKEY = 0x4000; // It is a Virtual Key Sequence
public static readonly VIRTUALCHARKEY = 0x8000; // Keyman 6.0: Virtual Key Cap Sequence NOT YET
// Note: OTHER_MODIFIER = 0x10000, used by KMX+ for the
// other modifier flag in layers, > 16 bit so not available here.
// See keys_mod_other in keyman_core_ldml.ts
public static readonly MASK_MODIFIER_CHIRAL = KMXFile.LCTRLFLAG | KMXFile.RCTRLFLAG | KMXFile.LALTFLAG | KMXFile.RALTFLAG;
public static readonly MASK_MODIFIER_SHIFT = KMXFile.K_SHIFTFLAG;
public static readonly MASK_MODIFIER_NONCHIRAL = KMXFile.K_CTRLFLAG | KMXFile.K_ALTFLAG;
@ -457,4 +461,4 @@ export class KMXFile {
throw "COMP_KEYBOARD size is "+this.COMP_KEYBOARD.size()+" but should be "+KMXFile.COMP_KEYBOARD_SIZE+" bytes";
}
}
}
}

9
common/web/types/src/util/util.ts
View file

@ -127,7 +127,7 @@ export function escapeRegexChar(ch: string) {
}
/** chars that must be escaped: syntax, C0 + C1 controls */
const REGEX_SYNTAX_CHAR = /^[\u0000-\u001F\u007F-\u009F{}\[\]\\?.^$*()/-]$/;
const REGEX_SYNTAX_CHAR = /^[\u0000-\u001F\u007F-\u009F{}\[\]\\?|.^$*()/-]$/;
function escapeRegexCharIfSyntax(ch: string) {
// escape if syntax or not valid
@ -148,6 +148,13 @@ function regexOne(hex: string): string {
// re-escape as 16 or 32 bit code units
return Array.from(unescaped).map(ch => escapeRegexCharIfSyntax(ch)).join('');
}
/**
* Escape a string (\uxxxx form) if there are any problematic codepoints
*/
export function escapeStringForRegex(s: string) : string {
return s.split('').map(ch => escapeRegexCharIfSyntax(ch)).join('');
}
/**
* Unescapes a string according to UTS#18§1.1, see <https://www.unicode.org/reports/tr18/#Hex_notation>
* @param s escaped string

4
common/windows/cpp/include/legacy_kmx_file.h
View file

@ -320,6 +320,10 @@
#define K_MODIFIERFLAG 0x007F
#define K_NOTMODIFIERFLAG 0xFF00 // I4548
// Note: OTHER_MODIFIER = 0x10000, used by KMX+ for the
// other modifier flag in layers, > 16 bit so not available here.
// See keys_mod_other in keyman_core_ldml.ts
/*
These sanity checks help ensure we don't
break on-disk struct sizes when we cross

3
common/windows/delphi/general/Upload_Settings.pas
View file

@ -94,6 +94,7 @@ uses
DebugPaths,
ErrorControlledRegistry,
RegistryKeys,
utilhttp,
VersionInfo;
const
@ -169,7 +170,7 @@ var
begin
if IsUpdate then IsUpdateInt := 1 else IsUpdateInt := 0;
Result := Format(URLPath_PackageDownload_Format, [PackageID, CKeymanVersionInfo.Tier, BCP47, IsUpdateInt]);
Result := Format(URLPath_PackageDownload_Format, [URLEncode(PackageID), URLEncode(CKeymanVersionInfo.Tier), URLEncode(BCP47), IsUpdateInt]);
end;
end.

4
common/windows/delphi/keyboards/kmxfileconsts.pas
View file

@ -129,6 +129,10 @@ const
KMX_ISVIRTUALKEY = $4000; // It is a Virtual Key Sequence
KMX_VIRTUALCHARKEY = $8000; // It is a virtual character key sequence - mnemonic layouts
// Note: KMX_OTHER_MODIFIER = $10000, used by KMX+ for the
// other modifier flag in layers, > 16 bit so not available here.
// See keys_mod_other in keyman_core_ldml.ts
// Combinations of key masks
KMX_MASK_MODIFIER_CHIRAL = KMX_LCTRLFLAG or KMX_RCTRLFLAG or KMX_LALTFLAG or KMX_RALTFLAG;
KMX_MASK_MODIFIER_SHIFT = KMX_SHIFTFLAG;

3
core/include/ldml/keyman_core_ldml.h
View file

@ -34,7 +34,7 @@
#define LDML_FINL_FLAGS_ERROR 0x1
#define LDML_KEYS_KEY_FLAGS_EXTEND 0x1
#define LDML_KEYS_KEY_FLAGS_GAP 0x2
#define LDML_KEYS_MOD_ALL 0x17F
#define LDML_KEYS_MOD_ALL 0x1017F
#define LDML_KEYS_MOD_ALT 0x40
#define LDML_KEYS_MOD_ALTL 0x4
#define LDML_KEYS_MOD_ALTR 0x8
@ -43,6 +43,7 @@
#define LDML_KEYS_MOD_CTRLL 0x1
#define LDML_KEYS_MOD_CTRLR 0x2
#define LDML_KEYS_MOD_NONE 0x0
#define LDML_KEYS_MOD_OTHER 0x10000
#define LDML_KEYS_MOD_SHIFT 0x10
#define LDML_LAYR_LIST_HARDWARE_TOUCH "touch"
#define LDML_LENGTH_BKSP 0xC

6
core/include/ldml/keyman_core_ldml.ts
View file

@ -270,6 +270,11 @@ class Constants {
*/
readonly keys_mod_shift = 0x0010;
/**
* bitmask for 'other'.
*/
readonly keys_mod_other = 0x10000;
/**
* Convenience map for modifiers
*/
@ -284,6 +289,7 @@ class Constants {
["ctrlL", this.keys_mod_ctrlL],
["ctrlR", this.keys_mod_ctrlR],
["shift", this.keys_mod_shift],
["other", this.keys_mod_other],
]
);

43
core/src/actions_normalize.cpp
View file

@ -119,39 +119,36 @@ bool km::core::actions_normalize(
normalization in our output, we now need to look for a normalization
boundary prior to the intersection of the cached_context and the output.
*/
if(!output.isEmpty()) {
while(n > 0 && !nfd->hasBoundaryBefore(output[0])) {
// The output may interact with the context further in normalization. We
// need to copy characters back further until we reach a normalization
// boundary.
while(n > 0 && output[0] && !nfd->hasBoundaryBefore(output[0])) {
// The output may interact with the context further in normalization. We
// need to copy characters back further until we reach a normalization
// boundary.
// Remove last code point from the context ...
// Remove last code point from the context ...
n = cached_context_string.moveIndex32(n, -1);
UChar32 chr = cached_context_string.char32At(n);
cached_context_string.remove(n);
n = cached_context_string.moveIndex32(n, -1);
UChar32 chr = cached_context_string.char32At(n);
cached_context_string.remove(n);
// And prepend it to the output ...
// And prepend it to the output ...
output.insert(0, chr);
// And finally remember that we now need to delete an additional NFD codepoint
actions.code_points_to_delete++;
output.insert(0, chr);
}
}
/*
At this point, our output and cached_context are coherent and normalization
will be complete at the edit boundary.
Now, we need to adjust the delete_back to match the number of characters
Now, we need to adjust the delete_back to match the number of codepoints
that must actually be deleted from the applications's NFU context
To adjust, we remove one character at a time from the app_context until
To adjust, we remove one codepoint at a time from the app_context until
its normalized form matches the cached_context normalized form.
*/
while(app_context_string.length()) {
while(app_context_string.countChar32()) {
icu::UnicodeString app_context_nfd;
nfd->normalize(app_context_string, app_context_nfd, icu_status);
assert(U_SUCCESS(icu_status));
@ -163,7 +160,13 @@ bool km::core::actions_normalize(
if(app_context_nfd.compare(cached_context_string) == 0) {
break;
}
app_context_string.remove(app_context_string.length()-1);
// remove the last UChar32
int32_t lastUChar32 = app_context_string.length()-1;
// adjust pointer to get the entire char (i.e. so we don't slice a non-BMP char)
lastUChar32 = app_context_string.getChar32Start(lastUChar32);
// remove the UChar32 (1 or 2 code units)
app_context_string.remove(lastUChar32);
nfu_to_delete++;
}
@ -319,4 +322,4 @@ bool km::core::actions_update_app_context_nfu(
delete [] items;
return status == KM_CORE_STATUS_OK;
}
}

4
core/src/km_core_action_api.cpp
View file

@ -56,7 +56,9 @@ void km::core::actions_dispose(
km_core_usv const *km::core::get_deleted_context(context const &app_context, unsigned int code_points_to_delete) {
auto p = app_context.end();
for(size_t i = code_points_to_delete; i > 0; i--, p--);
for(size_t i = code_points_to_delete; i > 0; i--, p--) {
assert(p != app_context.begin());
}
auto deleted_context = new km_core_usv[code_points_to_delete + 1];
for(size_t i = 0; i < code_points_to_delete; i++) {

7
core/src/km_core_state_context_set_if_needed.cpp
View file

@ -15,6 +15,7 @@
#include "state.hpp"
#include "debuglog.h"
#include "core_icu.h"
#include "kmx/kmx_xstring.h" // for Unicode routines
using namespace km::core;
@ -59,6 +60,12 @@ km_core_state_context_set_if_needed(
return KM_CORE_CONTEXT_STATUS_INVALID_ARGUMENT;
}
// if the app context begins with a trailing surrogate,
// skip over it.
if (Uni_IsSurrogate2(*new_app_context)) {
new_app_context++;
}
auto app_context = km_core_state_app_context(state);
auto cached_context = km_core_state_context(state);

1
core/src/kmx/kmx_plus.cpp
View file

@ -42,6 +42,7 @@ static_assert(LALTFLAG == LDML_KEYS_MOD_ALTL, "LDML modifier bitfield vs. kmx_fi
static_assert(K_ALTFLAG == LDML_KEYS_MOD_ALT, "LDML modifier bitfield vs. kmx_file.h #define mismatch");
static_assert(CAPITALFLAG == LDML_KEYS_MOD_CAPS, "LDML modifier bitfield vs. kmx_file.h #define mismatch");
static_assert(K_SHIFTFLAG == LDML_KEYS_MOD_SHIFT, "LDML modifier bitfield vs. kmx_file.h #define mismatch"); // "either" shift
// LDML_KEYS_MOD_OTHER is not present in kmx_file.h (>16 bit)
/**
* \def LDML_IS_VALID_MODIFIER_BITS test whether x is a valid modifier bitfield

23
core/src/ldml/C7043_ldml.md
View file

@ -461,17 +461,18 @@ For each key:
by the compiler.
- `mod`: 32-bit bitfield defined as below. Little endian values.
| Value | Meaning |`kmx_file.h` | Comment |
|----------|----------|---------------|---------------------------------------------|
| 0x0000 | `none` | | All zeros = no modifiers |
| 0x0001 | `ctrlL` | `LCTRLFLAG` | Left Control |
| 0x0002 | `ctrlR` | `RCTRLFLAG` | Right Control |
| 0x0004 | `altL` | `LALTFLAG` | Left Alt |
| 0x0008 | `altR` | `RALTFLAG` | Right Alt |
| 0x0010 | `shift` | `K_SHIFTFLAG` | Either Shift |
| 0x0020 | `ctrl` | `K_CTRLFLAG` | Either Control |
| 0x0040 | `alt` | `K_ALTFLAG` | Either Alt |
| 0x0100 | `caps` | `CAPITALFLAG` | Caps lock |
| Value | Meaning |`kmx_file.h` | Comment |
|----------|-----------|--------------------|-----------------------------------------------|
| 0x0000 | `none` | | All zeros = no modifiers |
| 0x0001 | `ctrlL` | `LCTRLFLAG` | Left Control |
| 0x0002 | `ctrlR` | `RCTRLFLAG` | Right Control |
| 0x0004 | `altL` | `LALTFLAG` | Left Alt |
| 0x0008 | `altR` | `RALTFLAG` | Right Alt |
| 0x0010 | `shift` | `K_SHIFTFLAG` | Either Shift |
| 0x0020 | `ctrl` | `K_CTRLFLAG` | Either Control |
| 0x0040 | `alt` | `K_ALTFLAG` | Either Alt |
| 0x0100 | `caps` | `CAPITALFLAG` | Caps lock |
| 0x10000 | `other` | n/a | Other (not used in conjunction with others) |
TODO-LDML: Note that conforming to other keyman values, left versus right shift
cannot be distinguished.

11
core/src/ldml/ldml_processor.cpp
View file

@ -377,6 +377,11 @@ ldml_event_state::emit_difference(const std::u32string &old_ctxt, const std::u32
// So the BBBBB needs to be removed and then CCC added.
auto ctxt_prefix = mismatch(old_ctxt.begin(), old_ctxt.end(), new_ctxt.begin(), new_ctxt.end());
// is the 'mismatch' at the end (i.e., no mismatch)?
if(ctxt_prefix.first == old_ctxt.end() && ctxt_prefix.second == new_ctxt.end()) {
return; // Optimization: We can just exit, there's nothing to do.
}
// handle a special case where we're simply changing from one marker to another.
// Example:
// 0. old_ctxtstr_changed ends with … U+FFFF U+0008 | U+0001 …
@ -390,14 +395,14 @@ ldml_event_state::emit_difference(const std::u32string &old_ctxt, const std::u32
// marker change.
// We can detect this because the unchanged_prefix will end with u+FFFF U+0008
//
// Oh, and yes, test case 'regex-test-8a-0' hits this.
// k_212* and k_213* hit this case.
std::u32string common_prefix(old_ctxt.begin(), ctxt_prefix.first);
if (common_prefix.length() >= 2) {
auto iter = common_prefix.rbegin();
if (*(iter++) == LDML_MARKER_CODE && *(iter++) == UC_SENTINEL) {
// adjust the iterator so that the "U+FFFF U+0008" is not a part of the common prefix.
ctxt_prefix.first -= 2;
ctxt_prefix.second += 2;
ctxt_prefix.first -= 2; // backup the 'mismatch' point to before the FFFF
ctxt_prefix.second -= 2; // backup the 'mismatch' point to before the FFFF
// Now, old_ctxtstr_changed and new_ctxtstr_changed will start with U+FFFF U+0008 …
}
}

11
core/src/ldml/ldml_vkeys.cpp
View file

@ -7,6 +7,7 @@
#include "ldml_vkeys.hpp"
#include "kmx_file.h"
#include <ldml/keyman_core_ldml.h>
namespace km {
namespace core {
@ -65,6 +66,16 @@ vkeys::lookup(km_core_virtual_key vk, uint16_t modifier_state, bool &found) cons
return ret;
}
}
// look for a layer with "other"
{
const vkey_id id_default(vk, (LDML_KEYS_MOD_OTHER));
ret = lookup(id_default, found);
if (found) {
return ret;
}
}
// default: return failure. found=false.
return ret;
}

19
core/tests/unit/kmnkbd/test_actions_normalize.cpp
View file

@ -152,7 +152,7 @@ void test_actions_normalize(
* this is initial_cached_context -
* actions_code_points_to_delete +
* actions_output) - no markers supported.
* If specified, final_cached_context_items
* If specified, final_cached_context_items
* must be nullptr.
* @param final_cached_context_items cached context _after_ actions have been
* applied -- NFU (essentially,
@ -440,6 +440,23 @@ void run_actions_normalize_tests() {
/* app_context: */ u"a\U0001F607bca\U0001F60E"
);
km_core_context_item items_11067[] = {
{ KM_CORE_CT_CHAR, {0,}, { U'𐒻' } },
{ KM_CORE_CT_CHAR, {0,}, { U'𐒷' } },
KM_CORE_CONTEXT_ITEM_END
};
// regression #11067
test_actions_normalize(
"A non-BMP char in context (#11067)",
/* app context pre transform: */ u"𐒻",
/* cached context post transform: */ u"𐒻𐒷",
/* cached context post transform: */ &items_11067[0],
/* action del, output: */ 0, U"𐒻𐒷",
// ---- results ----
/* action del, output: */ 1, U"𐒻𐒷",
/* app_context: */ u"𐒻𐒷"
);
}
void run_actions_update_app_context_nfu_tests() {

52
core/tests/unit/ldml/keyboards/k_007_transform_rgx-test.xml
View file

@ -88,4 +88,56 @@
<check result="E"/>
</test>
</tests>
<tests name="regression-11037">
<test name="caret">
<keystroke key="caret" />
<keystroke key="a" />
<check result="â"/>
</test>
<test name="pipe">
<keystroke key="pipe" />
<keystroke key="a" />
<check result="ą"/>
</test>
</tests>
<tests name="regression-11045-d">
<test name="caret">
<keystroke key="d" />
<keystroke key="name" />
<keystroke key="caret" />
<check result="caret" />
</test>
<test name="pipe">
<keystroke key="d" />
<keystroke key="name" />
<keystroke key="pipe" />
<check result="pipe" />
</test>
<test name="grave">
<keystroke key="d" />
<keystroke key="name" />
<keystroke key="grave" />
<check result="grave" />
</test>
</tests>
<tests name="regression-11045-i">
<test name="caret">
<keystroke key="i" />
<keystroke key="name" />
<keystroke key="caret" />
<check result="caret" />
</test>
<test name="pipe">
<keystroke key="i" />
<keystroke key="name" />
<keystroke key="pipe" />
<check result="pipe" />
</test>
<test name="grave">
<keystroke key="i" />
<keystroke key="name" />
<keystroke key="grave" />
<check result="grave" />
</test>
</tests>
</keyboardTest3>

17
core/tests/unit/ldml/keyboards/k_007_transform_rgx.xml
View file

@ -12,6 +12,9 @@ from https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-ke
<key id="zz" output="zz" />
<key id="zzz" output="zzz" />
<key id="zzzz" output="zzzz" />
<key id="caret" output="${caret}"/>
<key id="pipe" output="${pipe}"/>
<key id="name" output="\m{name}"/>
</keys>
<layers formId="us">
@ -26,7 +29,7 @@ from https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-ke
<row keys="grave 1 2 3 4 5 6 7 8 9 0" />
<row keys="Q W E R T Y U I O P" />
<row keys="A S D F G H J K L" />
<row keys="Z X C V zz zzz zzzz" />
<row keys="Z X C V zz zzz zzzz caret pipe name" />
<row keys="space" />
</layer>
</layers>
@ -34,6 +37,13 @@ from https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-ke
<variables>
<set id="upper" value="A E I O U YY" />
<set id="lower" value="a e i o u yy" />
<set id="direct" value="| ^ `"/>
<set id="indirect" value="${pipe} ${caret} ${grave}"/>
<set id="name" value="pipe caret grave"/>
<!-- for regression #11037 -->
<string id="caret" value="^" />
<string id="pipe" value="|" />
<string id="grave" value="`" />
</variables>
<transforms type="simple">
@ -45,6 +55,11 @@ from https://github.com/unicode-org/cldr/blob/keyboard-preview/docs/ldml/tr35-ke
</transformGroup>
<transformGroup>
<transform from="ez{2,3}" to="E" />
<transform from="${caret}a" to="â" />
<transform from="${pipe}a" to="ą" />
<!-- 'd' and 'i' should give the same result. -->
<transform from="d\m{name}($[direct])" to="$[1:name]"/>
<transform from="i\m{name}($[indirect])" to="$[1:name]"/>
</transformGroup>
</transforms>
</keyboard3>

27
core/tests/unit/ldml/keyboards/k_012_other.xml Normal file
View file

@ -0,0 +1,27 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
@@keys: [LALT K_BKQUOTE]
@@expected: \u0064
will match the default layer
-->
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info author="srl295" indicator="🙀" layout="qwerty" name="TestKbd"/>
<keys/>
<layers formId="us">
<layer id="base" modifiers="none">
<row keys="b" /> <!-- number row -->
</layer>
<layer id="shift" modifiers="shift">
<row keys="s" /> <!-- number row -->
</layer>
<layer id="other" modifiers="other">
<row keys="d" /> <!-- number row -->
</layer>
</layers>
</keyboard3>

12
core/tests/unit/ldml/keyboards/k_210_marker-test.xml
View file

@ -50,4 +50,16 @@
<check result="x" />
</test>
</tests>
<tests name="regression-11045">
<test name="regression-11045-grave">
<keystroke key="v" />
<keystroke key="grave"/>
<check result="`"/>
</test>
<test name="regression-11045-acute">
<keystroke key="v" />
<keystroke key="acute"/>
<check result="´"/>
</test>
</tests>
</keyboardTest3>

10
core/tests/unit/ldml/keyboards/k_210_marker.xml
View file

@ -24,12 +24,20 @@
<row keys="grave acute caret hacek" />
<row keys="q w e" /> <!-- etc -->
<row keys="a s d" /> <!-- etc -->
<row keys="z x c" /> <!-- etc -->
<row keys="z x c v" /> <!-- etc -->
</layer>
</layers>
<variables>
<!-- for regression #11045 -->
<set id="mark_accent" value="\m{acute} \m{grave}"/>
<set id="spacing_accent" value="´ `"/>
</variables>
<transforms type="simple">
<transformGroup>
<!-- for regression #11045 -->
<transform from="v($[mark_accent])" to="$[1:spacing_accent]" />
<transform from="\m{.}z" to="Z" />
</transformGroup>
<transformGroup>

23
core/tests/unit/ldml/keyboards/k_212_marker_11057-test.xml Normal file
View file

@ -0,0 +1,23 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE keyboardTest3 SYSTEM "../../../../../resources/standards-data/ldml-keyboards/45/dtd/ldmlKeyboardTest3.dtd">
<keyboardTest3 conformsTo="techpreview">
<info keyboard="k_212_marker_11057.xml" author="Team Keyboard" name="marker" />
<tests name="regression-11057">
<test name="caret-a">
<keystroke key="caret" />
<keystroke key="a" />
<check result="A" />
</test>
<test name="umlaut-e">
<keystroke key="umlaut" />
<keystroke key="e" />
<check result="E" />
</test>
<test name="crasher">
<keystroke key="caret"/>
<keystroke key="caret"/>
<keystroke key="umlaut"/>
<check result=""/>
</test>
</tests>
</keyboardTest3>

30
core/tests/unit/ldml/keyboards/k_212_marker_11057.xml Normal file
View file

@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="en" conformsTo="45">
<info author="srl295" indicator="🙀" layout="qwerty" name="k_212_regression test" />
<keys>
<key id="caret" output="\m{caret}" />
<key id="umlaut" output="\m{umlaut}" />
</keys>
<layers formId="us">
<layer modifiers="none" id="base">
<row keys="caret umlaut" />
<row keys="q w e r t y" />
<row keys="a" />
<row keys="z" />
<row keys="space" />
</layer>
</layers>
<transforms type="simple">
<transformGroup>
<transform from="\m{caret}a" to="A"/>
<transform from="\m{umlaut}e" to="E"/>
<!-- catch-all: drop any marker that didn't otherwise match before a char -->
<transform from="\m{.}(.)" to="$1" />
<!-- also drop any doubled marker -->
<transform from="\m{.}(\m{.})" to="$1" />
</transformGroup>
</transforms>
</keyboard3>

6
core/tests/unit/ldml/keyboards/meson.build
View file

@ -16,6 +16,7 @@ tests_from_cldr = [
'bn',
]
# these have 'embedded' (@@) testdata instead of a separate file
tests_without_testdata = [
# disabling 000 until we have updates to core or to the keyboard so that it passes
# 'k_000_null_keyboard',
@ -25,13 +26,13 @@ tests_without_testdata = [
'k_005_modbittest',
'k_010_mt',
'k_011_mt_iso',
'k_012_other',
'k_100_keytest',
'k_101_keytest',
'k_102_keytest',
]
# These tests have a k_001_tiny-test.xml file as well.
# These tests have a *-test.xml file as well.
tests_with_testdata = [
'ldml_test',
'k_001_tiny',
@ -44,6 +45,7 @@ tests_with_testdata = [
'k_201_reorder_esk',
'k_210_marker',
'k_211_marker_escape',
'k_212_marker_11057',
]
tests = tests_without_testdata

11
core/tests/unit/ldml/ldml_test_source.cpp
View file

@ -367,12 +367,19 @@ LdmlEmbeddedTestSource::vkey_to_event(std::string const &vk_event) {
modifier_state |= modifier;
} else {
vk = get_vk(s);
break;
if (vk == 0) {
std::cerr << "Error parsing [" << vk_event << "] - could not find vkey or modifier: " << s << std::endl;
}
assert(vk != 0);
break; // only one vkey allowed
}
}
// The string should be empty at this point
assert(!std::getline(f, s, ' '));
if (std::getline(f, s, ' ')) {
std::cerr << "Error parsing vkey ["<<vk_event<<"] - excess string after key: " << s << std::endl;
assert(false);
}
assert(vk != 0);
return {vk, modifier_state};

14
core/tests/unit/ldml/test_context_normalization.cpp
View file

@ -119,11 +119,23 @@ void test_context_normalization_invalid_unicode() {
teardown();
}
void test_context_normalization_lone_trailing_surrogate() {
// unpaired trail surrogate
km_core_cp const application_context[] = { 0xDC01, 0x0020, 0x0020, 0x0000 };
km_core_cp const cached_context[] = /* skipped*/ { 0x0020, 0x0020, 0x0000 };
setup("k_001_tiny.kmx");
assert(km_core_state_context_set_if_needed(test_state, application_context) == KM_CORE_CONTEXT_STATUS_UPDATED);
assert(is_identical_context(application_context+1, KM_CORE_DEBUG_CONTEXT_APP)); // first code unit is skipped
assert(is_identical_context(cached_context, KM_CORE_DEBUG_CONTEXT_CACHED));
teardown();
}
void test_context_normalization() {
test_context_normalization_already_nfd();
test_context_normalization_basic();
test_context_normalization_hefty();
// TODO: we need to strip illegal chars: test_context_normalization_invalid_unicode(); // -- unpaired surrogate, illegals
// TODO: see #10392 we need to strip illegal chars: test_context_normalization_invalid_unicode(); // -- unpaired surrogate, illegals
test_context_normalization_lone_trailing_surrogate();
}
//-------------------------------------------------------------------------------------

5
developer/src/common/include/kmn_compiler_errors.h
View file

@ -175,6 +175,11 @@
#define CERR_RepeatedBegin 0x00004073
#define CERR_VirtualKeyInContext 0x00004074
#define CERR_OutsTooLong 0x00004075
#define CERR_ExtendedStringTooLong 0x00004076
#define CERR_VirtualKeyExpansionTooLong 0x00004077
#define CERR_CharacterRangeTooLong 0x00004078
#define CWARN_TooManyWarnings 0x00002080
#define CWARN_OldVersion 0x00002081
#define CWARN_BitmapNotUsed 0x00002082

3
developer/src/kmc-keyboard-info/.gitignore vendored
View file

@ -1 +1,2 @@
src/imports/
src/imports/
test/fixtures/khmer_angkor/build/actual.keyboard_info

6
developer/src/kmc-keyboard-info/src/keyboard-info-compiler.ts
View file

@ -618,3 +618,9 @@ export class KeyboardInfoCompiler implements KeymanCompiler {
}
/**
* these are exported only for unit tests, do not use
*/
export const unitTestEndpoints = {
langtagsByTag,
};

1
developer/src/kmc-keyboard-info/test/fixtures/invalid-js-file/build/invalid_js_file.js vendored Normal file
View file

@ -0,0 +1 @@
// This is a file used to test the loadJsFile throws error if .js file is invalid error (see keyboard-info-compiler.ts)

BIN
developer/src/kmc-keyboard-info/test/fixtures/two-kmx/build/k_001___basic_input_unicodei.kmx vendored Normal file
View file

Binary file not shown.

BIN
developer/src/kmc-keyboard-info/test/fixtures/two-kmx/build/k_002___basic_input_unicode.kmx vendored Normal file
View file

Binary file not shown.

417
developer/src/kmc-keyboard-info/test/test-keyboard-info-compiler.ts
View file

@ -3,7 +3,11 @@ import { assert } from 'chai';
import 'mocha';
import { TestCompilerCallbacks } from '@keymanapp/developer-test-helpers';
import { makePathToFixture } from './helpers/index.js';
import { KeyboardInfoCompiler, KeyboardInfoCompilerResult } from '../src/keyboard-info-compiler.js';
import { KeyboardInfoCompiler, KeyboardInfoCompilerResult, unitTestEndpoints } from '../src/keyboard-info-compiler.js';
import langtags from "../src/imports/langtags.js";
import { KmpCompiler, KmpCompilerOptions } from '@keymanapp/kmc-package';
import { CompilerCallbacks, KMX, KeymanFileTypes, KeymanTargets, KmpJsonFile } from '@keymanapp/common-types';
import { KeyboardInfoFile, KeyboardInfoFilePlatform } from './keyboard-info-file.js';
const callbacks = new TestCompilerCallbacks();
@ -11,21 +15,58 @@ beforeEach(function() {
callbacks.clear();
});
const KHMER_ANGKOR_KPJ = makePathToFixture('khmer_angkor', 'khmer_angkor.kpj');
const KHMER_ANGKOR_JS = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.js');
const KHMER_ANGKOR_KPS = makePathToFixture('khmer_angkor', 'source', 'khmer_angkor.kps');
const KHMER_ANGKOR_KMP = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.kmp');
const KHMER_ANGKOR_SOURCES = {
kmpFilename: KHMER_ANGKOR_KMP,
sourcePath: 'release/k/khmer_angkor',
kpsFilename: KHMER_ANGKOR_KPS,
jsFilename: KHMER_ANGKOR_JS,
forPublishing: true,
};
const KHMER_ANGKOR_DISPLAY_FONT = "Mondulkiri-R.ttf";
const KHMER_ANGKOR_OSK_FONT = "khmer_busra_kbd.ttf";
const KHMER_ANGKOR_EXAMPLES_NO_ID = { keys: "x j m E r", text: "ខ្មែរ", note: "Name of language" };
const KHMER_ANGKOR_KEYBOARD = {
displayFont: KHMER_ANGKOR_DISPLAY_FONT,
oskFont: KHMER_ANGKOR_OSK_FONT,
name: "Khmer Angkor",
id: "khmer_angkor",
version: "1.3",
languages: [ { name: "Central Khmer (Khmer, Cambodia)", id: "km" } ],
examples: [ { id: "km", ...KHMER_ANGKOR_EXAMPLES_NO_ID } ]
};
const KHMER_ANGKOR_DISPLAY_FONT_INFO = { family: "Khmer Mondulkiri", source: [ KHMER_ANGKOR_DISPLAY_FONT ] };
const KHMER_ANGKOR_OSK_FONT_INFO = { family: "Khmer Busra Kbd", source: [ KHMER_ANGKOR_OSK_FONT ] };
const SECOND_DISPLAY_FONT = "second.ttf";
const SECOND_OSK_FONT = "second_osk.ttf";
const SECOND_EXAMPLES_NO_ID = { keys: "t w o", text: "two", note: "The number 2" };
const SECOND_KEYBOARD = {
displayFont: SECOND_DISPLAY_FONT,
oskFont: SECOND_OSK_FONT,
name: "Second Lang",
id: "second_lang",
version: "0.1",
languages: [ { name: "Second Language", id: "en" } ],
examples: [ { id: "en", ...SECOND_EXAMPLES_NO_ID } ]
};
const SECOND_DISPLAY_FONT_INFO = { family: "Second", source: [ SECOND_DISPLAY_FONT ] };
const SECOND_OSK_FONT_INFO = { family: "Second Kbd", source: [ SECOND_OSK_FONT ] };
describe('keyboard-info-compiler', function () {
it('compile a .keyboard_info file correctly', async function() {
const kpjFilename = makePathToFixture('khmer_angkor', 'khmer_angkor.kpj');
const jsFilename = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.js');
const kpsFilename = makePathToFixture('khmer_angkor', 'source', 'khmer_angkor.kps');
const kmpFilename = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.kmp');
const kpjFilename = KHMER_ANGKOR_KPJ;
const buildKeyboardInfoFilename = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.keyboard_info');
const sources = {
kmpFilename,
sourcePath: 'release/k/khmer_angkor',
kpsFilename,
jsFilename: jsFilename,
forPublishing: true,
};
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
@ -50,4 +91,354 @@ describe('keyboard-info-compiler', function () {
assert.deepEqual(actual, expected);
});
it('check preinit creates langtagsByTag correctly', async function() {
const compiler = new KeyboardInfoCompiler(); // indirectly call preinit()
assert.isNotNull(compiler);
const en_langtag = langtags.find(({ tag }) => tag === 'en');
assert.deepEqual((<any>unitTestEndpoints.langtagsByTag)['en'], en_langtag);
assert.deepEqual((<any>unitTestEndpoints.langtagsByTag)['en-Latn-US'], en_langtag);
assert.deepEqual((<any>unitTestEndpoints.langtagsByTag)['en-Latn'], en_langtag);
assert.deepEqual((<any>unitTestEndpoints.langtagsByTag)['en-US'], en_langtag);
});
it('check init initialises KeyboardInfoCompiler correctly', async function() {
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
assert.deepEqual(compiler['callbacks'], callbacks);
assert.deepEqual(compiler['options'], {sources});
});
it('check run returns null if KmpCompiler.init fails', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
const origKmpCompilerInit = KmpCompiler.prototype.init;
let result: KeyboardInfoCompilerResult;
try {
KmpCompiler.prototype.init = async (_callbacks: CompilerCallbacks, _options: KmpCompilerOptions): Promise<boolean> => false;
result = await compiler.run(kpjFilename, null);
} catch(e) {
assert.fail(e);
} finally {
KmpCompiler.prototype.init = origKmpCompilerInit;
}
assert.isNull(result);
});
it('check run returns null if KmpCompiler.transformKpsToKmpObject fails', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
const origKmpCompilerTransformKpsToKmpObject = KmpCompiler.prototype.transformKpsToKmpObject;
let result: KeyboardInfoCompilerResult;
try {
KmpCompiler.prototype.transformKpsToKmpObject = (_kpsFilename: string): KmpJsonFile.KmpJsonFile => null;
result = await compiler.run(kpjFilename, null);
} catch(e) {
assert.fail(e);
} finally {
KmpCompiler.prototype.transformKpsToKmpObject = origKmpCompilerTransformKpsToKmpObject;
}
assert.isNull(result);
});
it('check run returns null if loadJsFile fails', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const sources = KHMER_ANGKOR_SOURCES
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['loadJsFile'] = (_filename: string): string => null;
const result = await compiler.run(kpjFilename, null);
assert.isNull(result);
});
it('check run returns null if license is not MIT', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['isLicenseMIT'] = (_filename: string): boolean => false;
const result = await compiler.run(kpjFilename, null);
assert.isNull(result);
});
it('check run returns null if fillLanguages fails', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['fillLanguages'] = async (_kpsFilename: string, _keyboard_info: KeyboardInfoFile, _kmpJsonData: KmpJsonFile.KmpJsonFile): Promise<boolean> => false;
const result = await compiler.run(kpjFilename, null);
assert.isNull(result);
});
it('should write artifacts to disk', async function() {
const kpjFilename = KHMER_ANGKOR_KPJ;
const actualFilename = makePathToFixture('khmer_angkor', 'build', 'actual.keyboard_info');
const expectedFilename = makePathToFixture('khmer_angkor', 'build', 'khmer_angkor.keyboard_info');
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
const result = await compiler.run(kpjFilename, null);
assert.isNotNull(result);
if(fs.existsSync(actualFilename)) {
fs.rmSync(actualFilename);
}
result.artifacts.keyboard_info.filename = actualFilename;
assert.isTrue(await compiler.write(result.artifacts));
assert(fs.existsSync(actualFilename))
const actual = JSON.parse(fs.readFileSync(actualFilename, 'utf-8'));
const expected = JSON.parse(fs.readFileSync(expectedFilename, 'utf-8'));
// `lastModifiedDate` is dependent on time of run (not worth mocking)
delete actual['lastModifiedDate'];
delete expected['lastModifiedDate'];
assert.deepEqual(actual, expected);
if(fs.existsSync(actualFilename)) { // tidy up
fs.rmSync(actualFilename);
}
});
it('check mapKeymanTargetToPlatform returns correct platforms', async function() {
const compiler = new KeyboardInfoCompiler();
const map: {[index in KeymanTargets.KeymanTarget]: KeyboardInfoFilePlatform[]} = {
any: [],
androidphone: ['android'],
androidtablet: ['android'],
desktop: [],
ipad: ['ios'],
iphone: ['ios'],
linux: ['linux'],
macosx: ['macos'],
mobile: [],
tablet: [],
web: ['desktopWeb'],
windows: ['windows']
}
for (const [target, platform] of Object.entries(map)) {
assert.deepEqual(compiler['mapKeymanTargetToPlatform'](<KeymanTargets.KeymanTarget>target), platform);
}
});
it('check kmxFileVersionToString returns correct strings', async function() {
const compiler = new KeyboardInfoCompiler();
const convs = [
{num: 0x0000, str: '0.0'},
{num: 0x0001, str: '0.1'},
{num: 0x0100, str: '1.0'},
{num: 0x0101, str: '1.1'},
{num: 0x0A0A, str: '10.10'},
];
convs.forEach((conv) => {
assert.equal(compiler['kmxFileVersionToString'](conv.num), conv.str);
});
});
it('check loadKmxFiles returns empty array if .kmx file is missing from .kmp', async function() {
const kpsFilename = KHMER_ANGKOR_KPS;
const compiler = new KeyboardInfoCompiler();
const kmpCompiler = new KmpCompiler();
assert.isTrue(await kmpCompiler.init(callbacks, {}));
const kmpJsonData = kmpCompiler.transformKpsToKmpObject(kpsFilename);
assert.isNotNull(kmpJsonData);
// remove .kmx file
kmpJsonData.files = kmpJsonData.files.filter(file => !KeymanFileTypes.filenameIs(file.name, KeymanFileTypes.Binary.Keyboard));
const kmxFiles: {
filename: string,
data: KMX.KEYBOARD
}[] = compiler['loadKmxFiles'](kpsFilename, kmpJsonData);
assert.deepEqual(kmxFiles, []);
});
it('check loadKmxFiles throws error if .kmx file is missing from disk', async function() {
const kpsFilename = KHMER_ANGKOR_KPS;
const compiler = new KeyboardInfoCompiler();
const kmpCompiler = new KmpCompiler();
assert.isTrue(await kmpCompiler.init(callbacks, {}));
const kmpJsonData = kmpCompiler.transformKpsToKmpObject(kpsFilename);
assert.isNotNull(kmpJsonData);
// rename .kmx file in files list so it cannot be loaded from disk
const kmpIndex = kmpJsonData.files.findIndex(file => KeymanFileTypes.filenameIs(file.name, KeymanFileTypes.Binary.Keyboard));
kmpJsonData.files[kmpIndex].name = '../build/throw_error.kmx';
assert.throws(() => compiler['loadKmxFiles'](kpsFilename, kmpJsonData));
});
it('check loadKmxFiles can handle two .kmx files', async function() {
const jsFilename = makePathToFixture('two-kmx', 'build', 'two_kmx.js');
const kpsFilename = makePathToFixture('two-kmx', 'source', 'two_kmx.kps');
const kmpFilename = makePathToFixture('two-kmx', 'build', 'two_kmx.kmp');
const sources = {
kmpFilename,
sourcePath: 'release/k/two-kmx',
kpsFilename,
jsFilename: jsFilename,
forPublishing: true,
};
const kmx_filename_001 = 'k_001___basic_input_unicodei.kmx';
const kmx_filename_002 = 'k_002___basic_input_unicode.kmx';
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
const kmpJsonData: KmpJsonFile.KmpJsonFile = {
system: { fileVersion: '', keymanDeveloperVersion: '' },
options: null,
files: [
{ name: '../build/' + kmx_filename_001, description: 'Keyboard 001' },
{ name: '../build/' + kmx_filename_002, description: 'Keyboard 002' },
]
};
const kmxFiles: {
filename: string,
data: KMX.KEYBOARD
}[] = compiler['loadKmxFiles'](kpsFilename, kmpJsonData);
assert.equal(kmxFiles.length, 2);
assert.deepEqual(kmxFiles[0].filename, kmx_filename_001);
assert.deepEqual(kmxFiles[1].filename, kmx_filename_002);
assert.isNotNull(kmxFiles[0].data);
assert.isNotNull(kmxFiles[1].data);
});
it('check loadJsFile throws error if .js file is invalid', async function() {
const jsFilename = makePathToFixture('invalid-js-file', 'build', 'invalid_js_file.js');
const kpsFilename = makePathToFixture('invalid-js-file', 'source', 'invalid_js_file.kps');
const kmpFilename = makePathToFixture('invalid-js-file', 'build', 'invalid_js_file.kmp');
const sources = {
kmpFilename,
sourcePath: 'release/k/invalid-js-file',
kpsFilename,
jsFilename: jsFilename,
forPublishing: true,
};
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
const origTextDecoderDecode = TextDecoder.prototype.decode;
TextDecoder.prototype.decode = () => { throw new TypeError(); }
assert.throws(() => compiler['loadJsFile'](jsFilename));
TextDecoder.prototype.decode = origTextDecoderDecode;
});
it('check fillLanguages constructs keyboard_info.languages correctly', async function() {
const kmpJsonData: KmpJsonFile.KmpJsonFile = {
system: { fileVersion: '', keymanDeveloperVersion: '' },
options: null,
keyboards: [KHMER_ANGKOR_KEYBOARD],
};
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['fontSourceToKeyboardInfoFont'] = async (_kpsFilename: string, _kmpJsonData: KmpJsonFile.KmpJsonFile, _source: string[]) => {
if (_source[0] == KHMER_ANGKOR_DISPLAY_FONT) {
return KHMER_ANGKOR_DISPLAY_FONT_INFO;
} else { // osk font
return KHMER_ANGKOR_OSK_FONT_INFO;
}
}
const keyboard_info: KeyboardInfoFile = {};
const result = await compiler['fillLanguages'](KHMER_ANGKOR_KPS, keyboard_info, kmpJsonData);
assert.isTrue(result);
assert.deepEqual(keyboard_info.languages, {km: {
examples: [ KHMER_ANGKOR_EXAMPLES_NO_ID ],
font: KHMER_ANGKOR_DISPLAY_FONT_INFO,
oskFont: KHMER_ANGKOR_OSK_FONT_INFO,
languageName: "Khmer",
regionName: undefined,
scriptName: undefined,
displayName: "Khmer",
}});
});
it('check fillLanguages can handle two keyboards correctly', async function() {
const kmpJsonData: KmpJsonFile.KmpJsonFile = {
system: { fileVersion: '', keymanDeveloperVersion: '' },
options: null,
keyboards: [KHMER_ANGKOR_KEYBOARD, SECOND_KEYBOARD],
};
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
let callCount = 0;
compiler['fontSourceToKeyboardInfoFont'] = async (_kpsFilename: string, _kmpJsonData: KmpJsonFile.KmpJsonFile, _source: string[]) => {
callCount++;
const info = [KHMER_ANGKOR_DISPLAY_FONT_INFO, KHMER_ANGKOR_OSK_FONT_INFO, SECOND_DISPLAY_FONT_INFO, SECOND_OSK_FONT_INFO];
return info[callCount-1];
};
const keyboard_info: KeyboardInfoFile = {};
const result = await compiler['fillLanguages'](KHMER_ANGKOR_KPS, keyboard_info, kmpJsonData);
assert.isTrue(result);
assert.deepEqual(keyboard_info.languages, {km: {
examples: [ KHMER_ANGKOR_EXAMPLES_NO_ID ],
font: KHMER_ANGKOR_DISPLAY_FONT_INFO,
oskFont: KHMER_ANGKOR_OSK_FONT_INFO,
languageName: "Khmer",
regionName: undefined,
scriptName: undefined,
displayName: "Khmer",
}, en: {
examples: [ SECOND_EXAMPLES_NO_ID ],
font: SECOND_DISPLAY_FONT_INFO,
oskFont: SECOND_OSK_FONT_INFO,
languageName: "English",
regionName: undefined,
scriptName: undefined,
displayName: "English",
}});
});
it('check fillLanguages returns false if fontSourceToKeyboardInfoFont fails for display font', async function() {
const kmpJsonData: KmpJsonFile.KmpJsonFile = {
system: { fileVersion: '', keymanDeveloperVersion: '' },
options: null,
keyboards: [KHMER_ANGKOR_KEYBOARD],
};
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['fontSourceToKeyboardInfoFont'] = async (_kpsFilename: string, _kmpJsonData: KmpJsonFile.KmpJsonFile, _source: string[]) => {
if (_source[0] == KHMER_ANGKOR_DISPLAY_FONT) {
return null;
} else { // osk font
return KHMER_ANGKOR_OSK_FONT_INFO;
}
}
const result = await compiler['fillLanguages'](KHMER_ANGKOR_KPS, {}, kmpJsonData);
assert.isFalse(result);
});
it('check fillLanguages returns false if fontSourceToKeyboardInfoFont fails for osk font', async function() {
const kmpJsonData: KmpJsonFile.KmpJsonFile = {
system: { fileVersion: '', keymanDeveloperVersion: '' },
options: null,
keyboards: [KHMER_ANGKOR_KEYBOARD],
};
const sources = KHMER_ANGKOR_SOURCES;
const compiler = new KeyboardInfoCompiler();
assert.isTrue(await compiler.init(callbacks, {sources}));
compiler['fontSourceToKeyboardInfoFont'] = async (_kpsFilename: string, _kmpJsonData: KmpJsonFile.KmpJsonFile, _source: string[]) => {
if (_source[0] == KHMER_ANGKOR_DISPLAY_FONT) {
return KHMER_ANGKOR_DISPLAY_FONT_INFO;
} else { // osk font
return null;
}
}
const result = await compiler['fillLanguages'](KHMER_ANGKOR_KPS, {}, kmpJsonData);
assert.isFalse(result);
});
});

11
developer/src/kmc-kmn/src/compiler/compiler.ts
View file

@ -551,13 +551,19 @@ export class KmnCompiler implements KeymanCompiler, UnicodeSetParser {
return null;
}
// TODO-LDML: Catch OOM
if ((rangeCount * 2) < 0) {
throw new RangeError(`Internal error: negative rangeCount * 2 = ${rangeCount * 2}`);
}
const buf = this.wasmExports.malloc(rangeCount * 2 * Module.HEAPU32.BYTES_PER_ELEMENT);
if (buf <= 0) {
// out of memory will return zero.
throw new RangeError(`Internal error: wasm malloc() returned ${buf}`);
}
// fix \u1234 pattern format
pattern = KmnCompiler.fixNewPattern(pattern);
/** If <= 0: return code. If positive: range count */
const rc = Module.kmcmp_parseUnicodeSet(pattern, buf, rangeCount * 2);
if (rc >= 0) {
// If >= 0: it's a range count (which could be zero, an empty set).
const ranges = [];
const startu = (buf / Module.HEAPU32.BYTES_PER_ELEMENT);
for (let i = 0; i < rc; i++) {
@ -568,6 +574,7 @@ export class KmnCompiler implements KeymanCompiler, UnicodeSetParser {
this.wasmExports.free(buf);
return new UnicodeSet(pattern, ranges);
} else {
// rc is negative: it's an error code.
this.wasmExports.free(buf);
// translate error code into callback
this.callbacks.reportMessage(getUnicodeSetError(rc));

13
developer/src/kmc-kmn/src/compiler/kmn-compiler-messages.ts
View file

@ -559,6 +559,19 @@ export class KmnCompilerMessages {
static ERROR_VirtualKeyInContext = SevError | 0x074;
static Error_VirtualKeyInContext = () => m(this.ERROR_VirtualKeyInContext, `Virtual keys are not permitted in context`);
static ERROR_OutsTooLong = SevError | 0x075;
static Error_OutsTooLong = () => m(this.ERROR_OutsTooLong, `Store cannot be inserted with outs() as it makes the extended string too long`);
static ERROR_ExtendedStringTooLong = SevError | 0x076;
static Error_ExtendedStringTooLong = () => m(this.ERROR_ExtendedStringTooLong, `Extended string is too long`);
static ERROR_VirtualKeyExpansionTooLong = SevError | 0x077;
static Error_VirtualKeyExpansionTooLong = () => m(this.ERROR_VirtualKeyExpansionTooLong, `Virtual key expansion is too large`);
static ERROR_CharacterRangeTooLong = SevError | 0x078;
static Error_CharacterRangeTooLong = () => m(this.ERROR_CharacterRangeTooLong, `Character range is too large and cannot be expanded`);
static WARN_TooManyWarnings = SevWarn | 0x080;
static Warn_TooManyWarnings = () => m(this.WARN_TooManyWarnings, `Too many warnings or errors`);

15
developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_character_range_too_long.kmn vendored Normal file
View file

@ -0,0 +1,15 @@
store(&NAME) 'error_character_range_too_long'
store(&VERSION) '9.0'
begin unicode > use(main)
group(main) using keys
c maximum store length is 4096 UTF-16 code units, including U+0000 terminator
c #define GLOBAL_BUFSIZE 4096 // compfile.h
c so we need 0x101E - 0x0020 + 1 = 0x0FFF --> 4095 words
c See #11136 for calculation adjustment TODO
store(x) U+0020 .. U+101E
any(x) + 'x' > 'x' context

16
developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_extended_string_too_long.kmn vendored Normal file
View file

@ -0,0 +1,16 @@
store(&NAME) 'error_extended_string_too_long'
store(&VERSION) '9.0'
begin unicode > use(main)
group(main) using keys
c
c maximum store length is 4096 UTF-16 code units, including U+0000 terminator
c #define GLOBAL_BUFSIZE 4096 // compfile.h
c so we need 0x101B - 0x0020 + 1 = 0x0FFD --> 4092 words, + 4 = 4096 = too long
c See #11136 for calculation adjustment TODO
store(x) U+0020 .. U+101B
outs(x) 'abcd' + 'x' > 'x' context

18
developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_outs_too_long.kmn vendored Normal file
View file

@ -0,0 +1,18 @@
store(&NAME) 'error_outs_too_long'
store(&VERSION) '9.0'
begin unicode > use(main)
group(main) using keys
c maximum store length is 4096 UTF-16 code units, including U+0000 terminator
c #define GLOBAL_BUFSIZE 4096 // compfile.h
c so we need 0x101C - 0x0020 + 1 = 0x0FFD --> 4093 words
c + 1, for 'a' in the rule below = 4094, which triggers the buffer boundary check.
c Noting that this is conservative and losing 2 possible chars, but not fixing
c in compiler.cpp at this time.
c See #11136 for calculation adjustment TODO
store(x) U+0020 .. U+101C
'a' outs(x) + 'x' > 'x' context

17
developer/src/kmc-kmn/test/fixtures/invalid-keyboards/error_virtual_key_expansion_too_long.kmn vendored Normal file
View file

@ -0,0 +1,17 @@
store(&NAME) 'error_virtual_key_expansion_too_long'
store(&VERSION) '9.0'
begin unicode > use(main)
group(main) using keys
c maximum store length is 4096 UTF-16 code units, including U+0000 terminator
c #define GLOBAL_BUFSIZE 4096 // compfile.h
c so we need 0x101E - 0x0020 + 1 = 0x0FFF --> 4095 words
c each vk is 5 words long UC_SENTINEL CODE_EXTENDED shift key CODE_EXTENDEDEND (some long history here!)
c we start filling the buffer with 4066 words and then the remaining 30 bytes = 6 VKs A-F
c See #11136 for calculation adjustment TODO
store(x) U+0020 .. U+1000 [K_A] .. [K_F]
any(x) + 'x' > 'x' context

28
developer/src/kmc-kmn/test/test-messages.ts
View file

@ -94,4 +94,32 @@ describe('KmnCompilerMessages', function () {
assert.equal(callbacks.messages[0].message, "Virtual keys are not supported in output");
});
// ERROR_OutsTooLong
it('should generate ERROR_OutsTooLong if a store referenced in outs() is too long (more than GLOBAL_BUFSIZE elements)', async function() {
await testForMessage(this, ['invalid-keyboards', 'error_outs_too_long.kmn'], KmnCompilerMessages.ERROR_OutsTooLong);
// callbacks.printMessages();
});
// ERROR_ExtendedStringTooLong
it('should generate ERROR_ExtendedStringTooLong if an extended string is too long (more than GLOBAL_BUFSIZE elements)', async function() {
await testForMessage(this, ['invalid-keyboards', 'error_extended_string_too_long.kmn'], KmnCompilerMessages.ERROR_ExtendedStringTooLong);
// callbacks.printMessages();
});
// ERROR_VirtualKeyExpansionTooLong
it('should generate ERROR_VirtualKeyExpansionTooLong if a virtual key expansion is too long (more than GLOBAL_BUFSIZE elements)', async function() {
await testForMessage(this, ['invalid-keyboards', 'error_virtual_key_expansion_too_long.kmn'], KmnCompilerMessages.ERROR_VirtualKeyExpansionTooLong);
// callbacks.printMessages();
});
// ERROR_CharacterRangeTooLong
it('should generate ERROR_CharacterRangeTooLong if a character range would expand to be too long (more than GLOBAL_BUFSIZE elements)', async function() {
await testForMessage(this, ['invalid-keyboards', 'error_character_range_too_long.kmn'], KmnCompilerMessages.ERROR_CharacterRangeTooLong);
// callbacks.printMessages();
});
});

15
developer/src/kmc-ldml/src/compiler/keys.ts
View file

@ -487,7 +487,7 @@ export class KeysCompiler extends SectionCompiler {
sect: Keys,
hardware: string
): Keys {
const mod = translateLayerAttrToModifier(layer);
const mods = translateLayerAttrToModifier(layer);
const keymap = this.getKeymapFromForm(hardware);
// Iterate over rows (y) and cols (x) of the scancodes table.
@ -513,11 +513,14 @@ export class KeysCompiler extends SectionCompiler {
if (x < keys.length) {
key = keys[x];
}
sect.kmap.push({
vkey,
mod,
key, // key id, to be changed into key index at finalization
});
// push every combination
for (const mod of mods) {
sect.kmap.push({
vkey,
mod,
key, // key id, to be changed into key index at finalization
});
}
}
}
return sect;

35
developer/src/kmc-ldml/src/compiler/layr.ts
View file

@ -7,7 +7,6 @@ import { translateLayerAttrToModifier, validModifier } from '../util/util.js';
import DependencySections = KMXPlus.DependencySections;
import Layr = KMXPlus.Layr;
import LayrEntry = KMXPlus.LayrEntry;
import LayrList = KMXPlus.LayrList;
import LayrRow = KMXPlus.LayrRow;
@ -41,7 +40,7 @@ export class LayrCompiler extends SectionCompiler {
const { modifiers, id } = layer;
totalLayerCount++;
if (!validModifier(modifiers)) {
this.callbacks.reportMessage(CompilerMessages.Error_InvalidModifier({ modifiers, layer: id }));
this.callbacks.reportMessage(CompilerMessages.Error_InvalidModifier({ modifiers, layer: id || '' }));
valid = false;
}
});
@ -60,22 +59,28 @@ export class LayrCompiler extends SectionCompiler {
sect.lists = this.keyboard3.layers.map((layers) => {
const hardware = sections.strs.allocString(layers.formId);
// Already validated in validate
const layerEntries = [];
for (const layer of layers.layer) {
const rows = layer.row.map((row) => {
const erow: LayrRow = {
keys: row.keys.split(' ').map((id) => sections.strs.allocString(id)),
};
return erow;
});
const mods = translateLayerAttrToModifier(layer);
// push a layer entry for each modifier set
for (const mod of mods) {
layerEntries.push({
id: sections.strs.allocString(layer.id),
mod,
rows,
});
}
}
const list: LayrList = {
hardware,
minDeviceWidth: layers.minDeviceWidth || 0,
layers: layers.layer.map((layer) => {
const entry: LayrEntry = {
id: sections.strs.allocString(layer.id),
mod: translateLayerAttrToModifier(layer),
rows: layer.row.map((row) => {
const erow: LayrRow = {
keys: row.keys.split(' ').map((id) => sections.strs.allocString(id)),
};
return erow;
}),
};
return entry;
}),
layers: layerEntries,
};
return list;
});

29
developer/src/kmc-ldml/src/compiler/messages.ts
View file

@ -5,6 +5,9 @@ const SevWarn = CompilerErrorSeverity.Warn | CompilerErrorNamespace.LdmlKeyboard
const SevError = CompilerErrorSeverity.Error | CompilerErrorNamespace.LdmlKeyboardCompiler;
// const SevFatal = CompilerErrorSeverity.Fatal | CompilerErrorNamespace.LdmlKeyboardCompiler;
// sub-numberspace for transform errors
const SevErrorTransform = SevError | 0xF00;
/**
* @internal
*/
@ -171,13 +174,27 @@ export class CompilerMessages {
static Error_UnparseableReorderSet = (o: { from: string, set: string }) =>
m(this.ERROR_UnparseableReorderSet, `Illegal UnicodeSet "${def(o.set)}" in reorder "${def(o.from)}`);
static ERROR_UnparseableTransformFrom = SevError | 0x0029;
static Error_UnparseableTransformFrom = (o: { from: string, message: string }) =>
m(this.ERROR_UnparseableTransformFrom, `Invalid transfom from "${def(o.from)}": "${def(o.message)}"`);
// Available: 0x029
static ERROR_InvalidQuadEscape = SevError | 0x0030;
static Error_InvalidQuadEscape = (o: { cp: number }) =>
m(this.ERROR_InvalidQuadEscape, `Invalid escape "\\u${util.hexQuad(o?.cp || 0)}", use "\\u{${def(o?.cp?.toString(16))}}" instead.`);
m(this.ERROR_InvalidQuadEscape, `Invalid escape "\\u${util.hexQuad(o?.cp || 0)}". Hint: Use "\\u{${def(o?.cp?.toString(16))}}"`);
//
// Transform syntax errors begin at ...F00 (SevErrorTransform)
// This is a bit of a catch-all and represents messages bubbling up from the underlying regex engine
static ERROR_UnparseableTransformFrom = SevErrorTransform | 0x00;
static Error_UnparseableTransformFrom = (o: { from: string, message: string }) =>
m(this.ERROR_UnparseableTransformFrom, `Invalid transform from="${def(o.from)}": "${def(o.message)}"`);
static ERROR_IllegalTransformDollarsign = SevErrorTransform | 0x01;
static Error_IllegalTransformDollarsign = (o: { from: string }) =>
m(this.ERROR_IllegalTransformDollarsign, `Invalid transform from="${def(o.from)}": Unescaped dollar-sign ($) is not valid transform syntax.`,
'**Hint**: Use `\\$` to match a literal dollar-sign.');
static ERROR_TransformFromMatchesNothing = SevErrorTransform | 0x02;
static Error_TransformFromMatchesNothing = (o: { from: string }) =>
m(this.ERROR_TransformFromMatchesNothing, `Invalid transfom from="${def(o.from)}": Matches an empty string.`);
}

58
developer/src/kmc-ldml/src/compiler/tran.ts
View file

@ -139,25 +139,28 @@ export abstract class TransformCompiler<T extends TransformCompilerType, TranBas
let result = new TranTransform();
let cookedFrom = transform.from;
cookedFrom = sections.vars.substituteStrings(cookedFrom, sections);
// check for incorrect \uXXXX escapes. Do this before substituting markers or sets.
cookedFrom = this.checkEscapes(cookedFrom); // check for \uXXXX escapes before normalizing
cookedFrom = sections.vars.substituteStrings(cookedFrom, sections, true);
const mapFrom = VariableParser.CAPTURE_SET_REFERENCE.exec(cookedFrom);
const mapTo = VariableParser.MAPPED_SET_REFERENCE.exec(transform.to || '');
if (mapFrom && mapTo) { // TODO-LDML: error cases
result.mapFrom = sections.strs.allocString(mapFrom[1]); // var name
result.mapTo = sections.strs.allocString(mapTo[1]); // var name
} else {
result.mapFrom = sections.strs.allocString(''); // TODO-LDML
result.mapTo = sections.strs.allocString(''); // TODO-LDML
result.mapFrom = sections.strs.allocString('');
result.mapTo = sections.strs.allocString('');
}
if (cookedFrom === null) return null; // error
// the set substution will not produce raw markers `\m{...}` but they will already be in sentinel form.
cookedFrom = sections.vars.substituteSetRegex(cookedFrom, sections);
// add in markers. idempotent if no markers.
cookedFrom = sections.vars.substituteMarkerString(cookedFrom, true);
// check for incorrect \uXXXX escapes
cookedFrom = this.checkEscapes(cookedFrom); // check for \uXXXX escapes before normalizing
if (cookedFrom === null) return null; // error
// unescape from \u{} form to plain, or in some cases \uXXXX / \UXXXXXXXX for core
cookedFrom = util.unescapeStringToRegex(cookedFrom);
@ -169,12 +172,9 @@ export abstract class TransformCompiler<T extends TransformCompilerType, TranBas
cookedFrom = MarkerParser.nfd_markers(cookedFrom, true);
}
// Verify that the regex is syntactically valid
try {
new RegExp(cookedFrom, 'ug');
} catch (e) {
this.callbacks.reportMessage(CompilerMessages.Error_UnparseableTransformFrom({ from: transform.from, message: e.message }));
return null; // error
// perform regex validation
if (!this.isValidRegex(cookedFrom, transform.from)) {
return null;
}
// cookedFrom is cooked above, since there's some special treatment
@ -191,6 +191,38 @@ export abstract class TransformCompiler<T extends TransformCompilerType, TranBas
return result;
}
/**
* Validate the final regex
* @param cookedFrom the regex to use, missing the trailing '$'
* @param from the original from - for error reporting
* @returns true if OK
*/
private isValidRegex(cookedFrom: string, from: string) : boolean {
// check for any unescaped dollar sign here
if (/(?<!\\)(?:\\\\)*\$/.test(cookedFrom)) {
this.callbacks.reportMessage(CompilerMessages.Error_IllegalTransformDollarsign({ from }));
return false;
}
// Verify that the regex is syntactically valid
try {
const rg = new RegExp(cookedFrom + '$', 'ug');
// Tests against the regex:
// does it match an empty string?
if (rg.test('')) {
this.callbacks.reportMessage(CompilerMessages.Error_TransformFromMatchesNothing({ from }));
return false;
}
} catch (e) {
// We're exposing the internal regex error message here.
// In the future, CLDR plans to expose the EBNF for the transform,
// at which point we would have more precise validation prior to getting to this point.
this.callbacks.reportMessage(CompilerMessages.Error_UnparseableTransformFrom({ from, message: e.message }));
return false;
}
return true;
}
private compileReorderTranGroup(sections: DependencySections, reorders: LKReorder[]): TranGroup {
const result : TranGroup = {
type: constants.tran_group_type_reorder,

18
developer/src/kmc-ldml/src/compiler/vars.ts
View file

@ -192,6 +192,9 @@ export class VarsCompiler extends SectionCompiler {
validateSubstitutions(keyboard: LDMLKeyboard.LKKeyboard, st : Substitutions) : boolean {
keyboard?.variables?.string?.forEach(({value}) =>
st.markers.add(SubstitutionUse.variable, MarkerParser.allReferences(value)));
// get markers mentioned in a set
keyboard?.variables?.set?.forEach(({ value }) =>
VariableParser.setSplitter(value).forEach(v => st.markers.add(SubstitutionUse.match, MarkerParser.allReferences(v))));
return true;
}
@ -208,8 +211,6 @@ export class VarsCompiler extends SectionCompiler {
// first, strings.
variables?.string?.forEach((e) =>
this.addString(result, e, sections));
variables?.set?.forEach((e) =>
this.addSet(result, e, sections));
variables?.uset?.forEach((e) =>
this.addUnicodeSet(result, e, sections));
@ -222,6 +223,10 @@ export class VarsCompiler extends SectionCompiler {
const allMarkers : string[] = Array.from(mt.all).filter(m => m !== MarkerParser.ANY_MARKER_ID).sort();
result.markers = sections.list.allocList(allMarkers, {}, sections);
// sets need to be added late, because they can refer to markers
variables?.set?.forEach((e) =>
this.addSet(result, e, sections));
return result.valid() ? result : null;
}
@ -240,8 +245,13 @@ export class VarsCompiler extends SectionCompiler {
value = result.substituteStrings(value, sections);
// OK to do this as a substitute, because we've already validated the set above.
value = result.substituteSets(value, sections);
const items : string[] = VariableParser.setSplitter(value);
result.sets.push(new SetVarItem(id, items, sections));
// raw items - without marker substitution
const rawItems: string[] = VariableParser.setSplitter(value);
// cooked items - has substutition of markers
// this is not 'forMatch', all variables are to be assumed as string literals, not regex
// content.
const cookedItems: string[] = rawItems.map(v => result.substituteMarkerString(v, false));
result.sets.push(new SetVarItem(id, cookedItems, sections, rawItems));
}
addUnicodeSet(result: Vars, e: LDMLKeyboard.LKUSet, sections: DependencySections): void {
const { id } = e;

17
developer/src/kmc-ldml/src/util/util.ts
View file

@ -156,14 +156,19 @@ export function verifyValidAndUnique(
/**
* Determine modifier from layer info
* @param layer layer obj
* @returns modifier
* @returns modifier array
*/
export function translateLayerAttrToModifier(layer: LDMLKeyboard.LKLayer) : number {
export function translateLayerAttrToModifier(layer: LDMLKeyboard.LKLayer) : number[] {
const { modifiers } = layer;
if (!modifiers) return [constants.keys_mod_none];
return modifiers.split(',').map(m => translateModifierSubsetToLayer(m)).sort();
}
function translateModifierSubsetToLayer(modifiers: string) : number {
// TODO-LDML: Default #11072
if (modifiers) {
// TODO-LDML
if (modifiers.indexOf(',') !== -1) {
throw Error(`TODO-LDML #9838: ”,” in modifiers not supported yet.`);
throw Error(`translateModifierSubsetToLayer only takes a single subset of the modifiers`);
}
let mod = constants.keys_mod_none;
for (let str of modifiers.split(' ')) {
@ -183,8 +188,8 @@ export function translateLayerAttrToModifier(layer: LDMLKeyboard.LKLayer) : numb
export function validModifier(modifier?: string) : boolean {
if (!modifier) return true; // valid to have no modifier, == none
// TODO-LDML: enforce illegal combinations per spec.
for (let sub of modifier.split(',')) {
for (let str of sub.split(' ')) {
for (let sub of modifier.trim().split(',')) {
for (let str of sub.trim().split(' ')) {
if (!constants.keys_mod_map.has(str)) {
return false;
}

23
developer/src/kmc-ldml/test/fixtures/sections/keys/many-modifiers.xml vendored Normal file
View file

@ -0,0 +1,23 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info name="many-modifiers" />
<keys>
</keys>
<layers formId="iso">
<layer id="base" modifiers="none">
<row keys="a" />
</layer>
<layer modifiers="altR, ctrl shift">
<row keys="c" />
</layer>
<!--
for #11072
<layer modifiers="default">
<row keys="b" />
</layer>
-->
</layers>
</keyboard3>

20
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-IllegalTransformDollarsign-1.xml vendored Normal file
View file

@ -0,0 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info name="dollarsign-fail-1" />
<keys />
<variables>
<string id="markcaret" value="✔🥕" />
<set id="vowel" value="a i o"/>
<set id="caretVowel" value="🥕a 🥕i 🥕o"/>
</variables>
<transforms type="simple">
<transformGroup>
<transform from="${mark-caret}($[vowel])" to="$[1:caretVowel]" />
</transformGroup>
</transforms>
</keyboard3>

12
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-IllegalTransformDollarsign-2.xml vendored Normal file
View file

@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info name="dollarsign-fail-2" />
<keys />
<transforms type="simple">
<transformGroup>
<transform from="\\$" /> <!-- illegal -->
</transformGroup>
</transforms>
</keyboard3>

12
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-IllegalTransformDollarsign-3.xml vendored Normal file
View file

@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info name="dollarsign-fail-3" />
<keys />
<transforms type="simple">
<transformGroup>
<transform from="$" /> <!-- illegal -->
</transformGroup>
</transforms>
</keyboard3>

13
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-matches-nothing-1.xml vendored Normal file
View file

@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 conformsTo="45" xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" >
<info name="fail-matches-nothing"/>
<keys />
<transforms type="simple">
<transformGroup>
<transform from=""/>
</transformGroup>
</transforms>
</keyboard3>

13
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-matches-nothing-2.xml vendored Normal file
View file

@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 conformsTo="45" xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" >
<info name="fail-matches-nothing"/>
<keys />
<transforms type="simple">
<transformGroup>
<transform from="X{0,1}Y{0,1}"/>
</transformGroup>
</transforms>
</keyboard3>

13
developer/src/kmc-ldml/test/fixtures/sections/tran/fail-matches-nothing-3.xml vendored Normal file
View file

@ -0,0 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 conformsTo="45" xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" >
<info name="fail-matches-nothing"/>
<keys />
<transforms type="simple">
<transformGroup>
<transform from="X|"/>
</transformGroup>
</transforms>
</keyboard3>

12
developer/src/kmc-ldml/test/fixtures/sections/tran/ok-1.xml vendored Normal file
View file

@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<keyboard3 xmlns="https://schemas.unicode.org/cldr/45/keyboard3" locale="mt" conformsTo="45">
<info name="ok-1-dollarsign" />
<keys />
<transforms type="simple">
<transformGroup>
<transform from="\\\$" /> <!-- OK: dollarsign is escaped -->
</transformGroup>
</transforms>
</keyboard3>

19
developer/src/kmc-ldml/test/test-keys.ts
View file

@ -403,6 +403,25 @@ describe('keys.kmap', function () {
CompilerMessages.Error_MissingStringVariable({id: "varsok"}),
],
},
// modifiers test
{
// keep in sync with similar test in test-layr.ts
subpath: 'sections/keys/many-modifiers.xml',
callback(sect) {
const keys = <Keys> sect;
assert.ok(keys);
const { kmap } = keys;
const aMods = kmap.filter(({ key }) => key === 'a').map(({ mod }) => mod);
assert.sameDeepMembers(aMods, [
constants.keys_mod_none,
], 'modifiers for a');
const cMods = kmap.filter(({ key }) => key === 'c').map(({ mod }) => mod);
assert.sameDeepMembers(cMods, [
constants.keys_mod_altR,
constants.keys_mod_ctrl | constants.keys_mod_shift,
], 'modifiers for c');
},
},
], keysDependencies);
it('should reject layouts with too many hardware rows', async function() {

19
developer/src/kmc-ldml/test/test-layr.ts
View file

@ -110,5 +110,24 @@ describe('layr', function () {
subpath: 'sections/layr/invalid-missing-layer.xml',
errors: [CompilerMessages.Error_MustBeAtLeastOneLayerElement()],
},
{
// keep in sync with similar test in test-keys.ts
subpath: 'sections/keys/many-modifiers.xml',
callback(sect) {
const layr = <Layr> sect;
assert.ok(layr);
assert.equal(layr.lists.length, 1, 'layr.lists.length');
const layers = layr.lists[0];
const bymod = layers.layers.map(({id,mod,rows})=>([
id.value, mod, rows[0].keys[0].value,
]));
assert.sameDeepMembers(bymod, [
// flatten the layers for comparison, assume a single key
['base', constants.keys_mod_none, 'a'],
['', constants.keys_mod_altR, 'c'],
['', constants.keys_mod_ctrl | constants.keys_mod_shift, 'c'],
]);
},
},
]);
});

25
developer/src/kmc-ldml/test/test-tran.ts
View file

@ -334,6 +334,31 @@ describe('tran', function () {
CompilerMessages.Error_MissingStringVariable({ id: "missingstr" }),
],
},
// cases that share the same error code
...[1, 2].map(n => ({
subpath: `sections/tran/fail-IllegalTransformDollarsign-${n}.xml`,
errors: [
{
code: CompilerMessages.ERROR_IllegalTransformDollarsign,
matchMessage: /.*/,
}
],
})),
// successful compile
...[1].map(n => ({
subpath: `sections/tran/ok-${n}.xml`,
errors: false,
})),
// cases that share the same error code
...[1, 2, 3].map(n => ({
subpath: `sections/tran/fail-matches-nothing-${n}.xml`,
errors: [
{
code: CompilerMessages.ERROR_TransformFromMatchesNothing,
matchMessage: /.*/,
}
],
})),
// escaping
{
subpath: `sections/tran/tran-escape.xml`,

36
developer/src/kmc-ldml/test/test-utils.ts
View file

@ -174,29 +174,33 @@ describe('test of util/util.ts', () => {
});
describe('translateLayerAttrToModifier', () => {
it('should map from layer info to modifier number', () => {
assert.equal(translateLayerAttrToModifier({
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'base',
}), constants.keys_mod_none);
assert.equal(translateLayerAttrToModifier({
}), [constants.keys_mod_none]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'base',
modifiers: '',
}), constants.keys_mod_none);
assert.equal(translateLayerAttrToModifier({
}), [constants.keys_mod_none]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'base',
modifiers: 'none',
}), constants.keys_mod_none);
assert.equal(translateLayerAttrToModifier({
}), [constants.keys_mod_none]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'shift',
modifiers: 'shift',
}), constants.keys_mod_shift);
assert.equal(translateLayerAttrToModifier({
}), [constants.keys_mod_shift]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'shift',
modifiers: 'shift',
}), constants.keys_mod_shift);
assert.equal(translateLayerAttrToModifier({
}), [constants.keys_mod_shift]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'shiftOrCtrl',
modifiers: 'shift,ctrlL',
}), [constants.keys_mod_shift,constants.keys_mod_ctrlL]);
assert.sameDeepMembers(translateLayerAttrToModifier({
id: 'altshift',
modifiers: 'alt shift',
}), constants.keys_mod_alt | constants.keys_mod_shift);
}), [constants.keys_mod_alt | constants.keys_mod_shift]);
});
it('should round trip each possible modifier', () => {
for(let str of constants.keys_mod_map.keys()) {
@ -204,8 +208,8 @@ describe('test of util/util.ts', () => {
id: str,
modifiers: `${str}`,
};
assert.equal(translateLayerAttrToModifier(layer),
constants.keys_mod_map.get(str), str);
assert.sameDeepMembers(translateLayerAttrToModifier(layer),
[constants.keys_mod_map.get(str)], str);
}
});
it('should round trip each possible modifier with altL', () => {
@ -214,8 +218,8 @@ describe('test of util/util.ts', () => {
id: str,
modifiers: `${str} altL`,
};
assert.equal(translateLayerAttrToModifier(layer),
constants.keys_mod_map.get(str) | constants.keys_mod_altL, str);
assert.sameDeepMembers(translateLayerAttrToModifier(layer),
[constants.keys_mod_map.get(str) | constants.keys_mod_altL], str);
}
});
});

1
developer/src/kmc-model/build.sh
View file

@ -15,6 +15,7 @@ cd "$THIS_SCRIPT_PATH"
builder_describe "Build Keyman kmc Lexical Model Compiler module" \
"@/common/web/keyman-version" \
"@/developer/src/common/web/test-helpers" \
"@/common/models/templates test" \
"clean" \
"configure" \
"build" \

1
developer/src/kmc-model/package.json
View file

@ -37,6 +37,7 @@
},
"devDependencies": {
"@keymanapp/developer-test-helpers": "*",
"@keymanapp/models-templates": "*",
"@types/chai": "^4.1.7",
"@types/mocha": "^5.2.7",
"@types/node": "^20.4.1",

46
developer/src/kmc-model/test/test-compile-trie.ts
View file

@ -1,11 +1,13 @@
import { LexicalModelCompiler } from '../src/lexical-model-compiler.js';
import {assert} from 'chai';
import fs from 'fs';
import 'mocha';
import {makePathToFixture, compileModelSourceCode} from './helpers/index.js';
import { createTrieDataStructure } from '../src/build-trie.js';
import { ModelCompilerError } from '../src/model-compiler-messages.js';
import { TestCompilerCallbacks } from '@keymanapp/developer-test-helpers';
import { TrieModel } from '@keymanapp/models-templates';
describe('LexicalModelCompiler', function () {
const callbacks = new TestCompilerCallbacks();
@ -165,8 +167,48 @@ describe('createTrieDataStructure()', function () {
it('does not create `null`/"undefined"-keyed children', function () {
const WORDLIST_FILENAME = makePathToFixture('example.qaa.wordlist-ordering', 'wordlist.tsv');
// check to ensure the Trie is fully well-formed.
//
// Is pretty much a JSON-encoded Trie spec, stringifying { root: Node, totalWeight: number }
// as used to initialize the Trie.
let sourceCode = createTrieDataStructure([WORDLIST_FILENAME], (wf) => wf);
assert.notMatch(sourceCode, /"undefined"/);
// Simple first-pass check: the signs of #11073 are not present.
assert.notMatch(sourceCode, /undefined/);
assert.notMatch(sourceCode, /null/);
})
// A more complex check: load and use the resulting TrieModel to check that all
// words are accessible via `.predict`.
// First, load the model.
const trieSpec = JSON.parse(sourceCode);
const model = new TrieModel(trieSpec);
assert.isOk(model);
// Gets the list of all words in the wordlist fixture.
const rawWordlist = fs.readFileSync(WORDLIST_FILENAME).toString();
const words = rawWordlist.split('\n').map((line) => {
const columns = line.split('\t')
if(!columns.length || !columns[0]) {
return undefined;
} else {
return columns[0];
}
}).filter((entry) => entry !== undefined);
// We'll track all _observed_ words from the model here.
const set = new Set<string>();
const contextFromWord = (word: string) => {
return {left: word, startOfBuffer: true, endOfBuffer: true};
};
// Using each word as a prediction prefix, attempt to get a suggestion corresponding to each.
for(let word of words) {
const rawSuggestions = model.predict({insert: '', deleteLeft: 0}, contextFromWord(word));
const suggestions = rawSuggestions.map((entry) => entry.sample.displayAs);
suggestions.forEach((suggestion) => set.add(suggestion));
}
// The actual assertion: did we see each word as a suggestion?
assert.sameMembers([...set], words, "Could not suggest all words in the wordlist");
});
});

2
developer/src/kmc/build.sh
View file

@ -126,7 +126,7 @@ if builder_start_action bundle; then
mkdir -p build/dist
node build-bundler.js
./node_modules/.bin/sentry-cli sourcemaps inject \
sentry-cli sourcemaps inject \
--org keyman \
--project keyman-developer \
--release "$VERSION_GIT_TAG" \

2
developer/src/kmc/package.json
View file

@ -55,7 +55,7 @@
"build/unicode-license.txt"
],
"devDependencies": {
"@sentry/cli": "^2.19.4",
"@sentry/cli": "^2.31.0",
"@types/chai": "^4.1.7",
"@types/mocha": "^5.2.7",
"@types/node": "^20.4.1",

4
developer/src/kmcmplib/src/CompMsg.cpp
View file

@ -111,6 +111,10 @@ const struct CompilerError CompilerErrors[] = {
{ CERR_DuplicateStore , "A store with this name has already been defined."},
{ CERR_RepeatedBegin , "Begin has already been set"},
{ CERR_VirtualKeyInContext , "Virtual keys are not permitted in context"},
{ CERR_OutsTooLong , "Store cannot be inserted with outs() as it makes the extended string too long" },
{ CERR_ExtendedStringTooLong , "Extended string is too long" },
{ CERR_VirtualKeyExpansionTooLong , "Virtual key expansion is too large" },
{ CERR_CharacterRangeTooLong , "Character range is too large and cannot be expanded" },
{ CHINT_UnreachableRule , "This rule will never be matched as another rule takes precedence"},
{ CHINT_NonUnicodeFile , "Keyman Developer has detected that the file has ANSI encoding. Consider converting this file to UTF-8"},

27
developer/src/kmcmplib/src/Compiler.cpp
View file

@ -1827,6 +1827,12 @@ KMX_DWORD GetXStringImpl(PKMX_WCHAR tstr, PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX
p = str;
do
{
if (mx >= max) {
// This is an error condition, we want the compiler
// to crash if we reach this
return CERR_BufferOverflow;
}
tokenFound = FALSE;
while (iswspace(*p) && !u16chr(token, *p)) p++;
if (!*p) break;
@ -1905,7 +1911,7 @@ KMX_DWORD GetXStringImpl(PKMX_WCHAR tstr, PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX
case 1:
q = (PKMX_WCHAR) u16chr(p + 1, '\"');
if (!q) return CERR_UnterminatedString;
if ((int)(q - p) - 1 + mx > max) return CERR_UnterminatedString;
if ((int)(q - p) - 1 + mx > max) return CERR_ExtendedStringTooLong;
if (sFlag) return CERR_StringInVirtualKeySection;
u16ncat(tstr, p + 1, (int)(q - p) - 1); // I3481
mx += (int)(q - p) - 1;
@ -1915,7 +1921,7 @@ KMX_DWORD GetXStringImpl(PKMX_WCHAR tstr, PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX
case 2:
q = (PKMX_WCHAR) u16chr(p + 1, '\'');
if (!q) return CERR_UnterminatedString;
if ((int)(q - p) - 1 + mx > max) return CERR_UnterminatedString;
if ((int)(q - p) - 1 + mx > max) return CERR_ExtendedStringTooLong;
if (sFlag) return CERR_StringInVirtualKeySection;
u16ncat(tstr, p + 1, (int)(q - p) - 1); // I3481
mx += (int)(q - p) - 1;
@ -2031,7 +2037,9 @@ KMX_DWORD GetXStringImpl(PKMX_WCHAR tstr, PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX
for (q = fk->dpStoreArray[i].dpString; *q; q++)
{
tstr[mx++] = *q;
if (mx >= max - 1) return CERR_BufferOverflow;
if (mx >= max - 1) {
return CERR_OutsTooLong;
}
}
tstr[mx] = 0;
continue;
@ -2421,7 +2429,6 @@ KMX_DWORD GetXStringImpl(PKMX_WCHAR tstr, PFILE_KEYBOARD fk, PKMX_WCHAR str, KMX
ErrChr = 0;
return CERR_None;
}
if (mx >= max) return CERR_BufferOverflow;
} while (*p);
if (!*token)
@ -2630,7 +2637,9 @@ KMX_DWORD process_expansion(PFILE_KEYBOARD fk, PKMX_WCHAR q, PKMX_WCHAR tstr, in
return CERR_ExpansionMustBePositive;
}
// Verify space in buffer
if (*mx + (HighKey - BaseKey) * 5 + 1 >= max) return CERR_BufferOverflow;
if (*mx + (HighKey - BaseKey) * 5 + 1 >= max) {
return CERR_VirtualKeyExpansionTooLong;
}
// Inject an expansion.
for (BaseKey++; BaseKey < HighKey; BaseKey++) {
// < HighKey because caller will add HighKey to output
@ -2657,12 +2666,16 @@ KMX_DWORD process_expansion(PFILE_KEYBOARD fk, PKMX_WCHAR q, PKMX_WCHAR tstr, in
// < HighChar because caller will add HighChar to output
if (Uni_IsSMP(BaseChar)) {
// We'll test on each char to avoid complex calculations crossing SMP boundary
if (*mx + 3 >= max) return CERR_BufferOverflow;
if (*mx + 3 >= max) {
return CERR_CharacterRangeTooLong;
}
tstr[(*mx)++] = (KMX_WCHAR) Uni_UTF32ToSurrogate1(BaseChar);
tstr[(*mx)++] = (KMX_WCHAR) Uni_UTF32ToSurrogate2(BaseChar);
}
else {
if (*mx + 2 >= max) return CERR_BufferOverflow;
if (*mx + 2 >= max) {
return CERR_CharacterRangeTooLong;
}
tstr[(*mx)++] = (KMX_WCHAR) BaseChar;
}
}

3
developer/src/kmconvert/kmconvert.dpr
View file

@ -88,7 +88,8 @@ uses
Keyman.System.LexicalModelUtils in '..\common\delphi\lexicalmodels\Keyman.System.LexicalModelUtils.pas',
KeymanDeveloperOptions in '..\tike\main\KeymanDeveloperOptions.pas',
Keyman.Developer.System.KeymanDeveloperPaths in '..\tike\main\Keyman.Developer.System.KeymanDeveloperPaths.pas',
Keyman.Developer.System.LdmlKeyboardProjectTemplate in 'Keyman.Developer.System.LdmlKeyboardProjectTemplate.pas';
Keyman.Developer.System.LdmlKeyboardProjectTemplate in 'Keyman.Developer.System.LdmlKeyboardProjectTemplate.pas',
utilhttp in '..\..\..\common\windows\delphi\general\utilhttp.pas';
{$R icons.RES}
{$R version.res}

1
developer/src/kmconvert/kmconvert.dproj
View file

@ -195,6 +195,7 @@
<DCCReference Include="..\tike\main\KeymanDeveloperOptions.pas"/>
<DCCReference Include="..\tike\main\Keyman.Developer.System.KeymanDeveloperPaths.pas"/>
<DCCReference Include="Keyman.Developer.System.LdmlKeyboardProjectTemplate.pas"/>
<DCCReference Include="..\..\..\common\windows\delphi\general\utilhttp.pas"/>
<BuildConfiguration Include="Release">
<Key>Cfg_2</Key>
<CfgParent>Base</CfgParent>

3
developer/src/setup/setup.dpr
View file

@ -25,7 +25,8 @@ uses
utilexecute in '..\..\..\common\windows\delphi\general\utilexecute.pas',
KeymanVersion in '..\..\..\common\windows\delphi\general\KeymanVersion.pas',
SFX in '..\..\..\common\windows\delphi\setup\SFX.pas',
Keyman.System.UpdateCheckResponse in '..\..\..\common\windows\delphi\general\Keyman.System.UpdateCheckResponse.pas';
Keyman.System.UpdateCheckResponse in '..\..\..\common\windows\delphi\general\Keyman.System.UpdateCheckResponse.pas',
utilhttp in '..\..\..\common\windows\delphi\general\utilhttp.pas';
{$R icons.res}
{$R version.res}

1
developer/src/setup/setup.dproj
View file

@ -124,6 +124,7 @@
<DCCReference Include="..\..\..\common\windows\delphi\general\KeymanVersion.pas"/>
<DCCReference Include="..\..\..\common\windows\delphi\setup\SFX.pas"/>
<DCCReference Include="..\..\..\common\windows\delphi\general\Keyman.System.UpdateCheckResponse.pas"/>
<DCCReference Include="..\..\..\common\windows\delphi\general\utilhttp.pas"/>
<BuildConfiguration Include="Debug">
<Key>Cfg_2</Key>
<CfgParent>Base</CfgParent>

6
developer/src/tike/xml/layoutbuilder/builder.js
View file

@ -18,8 +18,8 @@ $(function() {
this.saveSelection = function() {
let key = builder.selectedKey(), subKey = builder.selectedSubKey();
return {
id: key ? $(key).data('id') : null,
subId: subKey ? $(subKey).data('id') : null
id: key.length ? $(key).data('id') : null,
subId: subKey.length ? $(subKey).data('id') : null
};
}
@ -971,7 +971,7 @@ $(function() {
$('#kbd-scroll-container').on('scroll', function () {
const key = builder.selectedKey();
if(key) {
if(key.length) {
builder.moveWedgesAround(key[0]);
}
});

4
developer/src/tools/sentry-upload-difs.sh
View file

@ -65,7 +65,7 @@ sourcemap_paths=(
)
echo "Uploading symbols for developer/"
./src/kmc/node_modules/.bin/sentry-cli upload-dif \
sentry-cli upload-dif \
--project keyman-developer \
--include-sources \
--no-zips \
@ -73,7 +73,7 @@ echo "Uploading symbols for developer/"
upload_sourcemap() {
local smpath="$1"
"$KEYMAN_ROOT/developer/src/kmc/node_modules/.bin/sentry-cli" sourcemaps upload \
sentry-cli sourcemaps upload \
--no-dedupe \
--org keyman \
--project keyman-developer \

1
ios/engine/KMEI/KeymanEngine/Classes/Constants.swift
View file

@ -61,6 +61,7 @@ public enum Key {
// Settings-related keys
static let optShouldReportErrors = "ShouldReportErrors"
// Deprecated - no longer used
static let optShouldShowBanner = "ShouldShowBanner"
static let optSpacebarText = "SpacebarText"
// This one SHOULD be app-only, but is needed by the currently

Some files were not shown because too many files have changed in this diff Show more