5.9 KiB
Gesture Input Serialization
The GestureRecognizer engine is designed to handle and process asynchronous, continuous input of one or more touchpoints at the same time. The processing of events takes time, though, and on older devices it is possible for new input events to come in while older events are still being processed. As it is possible for an earlier event to affect how a later event should be processed, it is necessary to ensure that all events are processed in a specific order. We wish to capture the user's original intent for each aspect of input and later match it against the system state that a user would reasonably anticipate for the input based on their previous interactions, even if such a state hasn't yet actually been reached or activated at the true time of input.
For a keyboard, suppose a user taps the Shift and A keys with minimal time gap between the two events. The Shift state may not be in place yet once the A is tapped - event processing does take some time, after all. The serialization scheme used by the engine will delay processing for the A until the Shift state has taken effect, leading a capital A to be emitted instead of a lowercase a.
This serialization scheme used by the gesture engine is further complicated by the feature allowing a gesture to be resolved when receiving a new input without associating that input with the gesture. For example, gestures can be specified such that one tap's gesture instantly finalizes whenver newly-incoming second tap is received. Should finalization of the former trigger a change in engine state - say, to model a change in layer - it is possible that the newly-incoming input should not be processed further until the former input is fully processed. Furthermore, the engine leverages Promises internally to help keep the different layers of gesture processing separate and isolated; these need a chance to resolve before continuing processing of the new event and any events that may follow it.
Gesture Processing serialization
The first point that requires special handling is detection of newly-received inputs - new mouse or touch interaction points. As some gesture models may specify autocompletion whenever a new touch comes in, those should resolve before further handling the new input.
Toward this end, MatcherSelector's matchGesture method maintains a Promise serving as a lock within the field pendingMatchSetup. This is set whenever checking such cases - whether or not existing gestures have special interactions triggered by new touchpoints - and is cleared once these checks have completed. This lock is referenced early within the same method to prevent matchGesture from handling any subsequent inputs while waiting on this process to complete.
The method calling this - TouchpointCoordinator.onNewTrackedPath - also includes special handling for this case. It will not emit an event for the newly-incoming touchpoint until after its matchGesture call has completed and determined which set of gestures the input may match. Said process relates strongly to the serialization scheme documented below and is the point where these two aspects interact. The event raised by onNewTrackedPath - 'inputstart' - should only fire once gesture processing may proceed for the input.
Serialization at this level may be further complicated for cases where a gesture state was being maintained due to having pushed an engine alt-state that had at least one live gesture; should the last sustaining sub-gesture reach completion by the new input, the sustain state must be unwound before continuing. For example, suppose a new menu appears when a key or button is held down, with this action triggering an "alt-mode" using a different base set of gesture models for use of that new menu while it displays. That base gesture may specify that it is to be kept alive to avoid interrupting interactions with the new menu when the "held down" key is released, only dropping the menu (and the "alt-mode") after those interactions resolve.
Touchpath sampling serialization
Delaying new inputs also invokes a need to delay events that occur at or after the same timestamp as the first delayed input. After all, tracking for any new input, including for the first delayed input, needs to be initiated before any future observations for such an input may be properly handled. This is especially critical for touch processing, as it is easily possible for the user to perform touch input with multiple fingers in parallel.
To address this, the touch-event processing component - TouchEventEngine - captures events as they come in and adds them to a processing queue, implemented as AsyncClosureDispatchQueue, that ensures the events stay in proper sequential, serialized order by use of Promises. It also creates specialized Promise-based locks whenever a new touchpoint is received that are added to the serialization queue; these locks are only released when TouchpointCoordinator.onNewTrackedPath has proceeded far enough to publish the touchpoint's tracking GestureSource instance - this is the purpose of the fulfillInputStart call within onNewTrackedPath.
These locks are important in case external code wishes to receive each incoming input event for the touchpoint via event; path event listeners can't be registered without access to the base touchpath object.
Use of the AsyncClosureDispatchQueue comes with an added benefit; it enforces a macrotask queue wait after gesture-level processing completes for each input event. This gives newly-incoming input events a chance to be recorded in a responsive fashion, which is particularly important for any gestures that require a sense of timing. This way, the true, original time of the input events will stay reasonably accurate, even if the actual gesture-level processing operations are delayed until later.