6.7 KiB
Gesture Input Serialization
The GestureRecognizer engine is designed to handle and process asynchronous, continuous input of one or more touchpoints in parallel. The processing of events takes time, though, and on older devices it is possible for new input events to come in while older events are still being processed. As it is possible for an earlier event to affect how a later event should be processed, it is necessary to ensure that all events are processed in a specific order. We wish to capture the user's original intent for each aspect of input and later match it against the system state that a user would reasonably anticipate for the input based on their previous interactions, even if such a state hasn't yet actually been reached or activated at the true time of input.
For a keyboard, suppose a user taps the Shift and A keys with minimal time gap between the two events. Also suppose that external code sets state = 'shift' when processing the Shift key, with default its current value - perhaps reflecting the name of different layers in an on-screen keyboard. For this example, the Shift state may not be in place yet once the A is tapped - event processing does take some time, after all - especially if the events are considered simultaneous by the browser.
Note that JS mouse and touch events lock in their target elements at the time of observation, even if that element is swapped out of the DOM by the time the event is actually handled. The serialization scheme used by the engine handles the As event (provisionally selecting the unshifted A, per the original default state) but delays further processing for it until the Shift state has taken effect. If supplied with a properly-designed itemIdentifier method as part of the engine's configuration, the serialization processes then uses that method to finalize the associated element once processing resumes, remapping from the unshifted A to the Shift-layer A per the shift in the event's associated engine state.
Furthermore, the engine leverages Promises internally to help keep the different layers of gesture processing separate and isolated; these need a chance to resolve before continuing processing of the new event and any events that may follow it. In particular, cases like the Shift + A example invoke setTimeout waits that leverage the JS task queue, not just the microtask queue that Promises natively leverage. During times of rapid, heavy input, it is thus possible for newly-incoming touch events to be received before all Promises for prior input gesture processing have fully resolved. It is thus necessary to artificially defer processing of newly-incoming touch events so that their processing does not occur before cases like Shift + A above fully resolve.
Gesture Processing serialization
The first point that requires special handling is detection of newly-received inputs - new mouse or touch interaction points. As some gesture models may specify autocompletion whenever a new touch comes in, those should resolve before further handling the new input.
Toward this end, MatcherSelector's matchGesture method maintains a Promise serving as a lock within the field pendingMatchSetup. This is set whenever checking such cases - whether or not existing gestures have special interactions triggered by new touchpoints - and is cleared once these checks have completed. This lock is referenced early within the same method to prevent matchGesture from handling any subsequent inputs while waiting on this process to complete.
The method calling this - TouchpointCoordinator.onNewTrackedPath - also includes special handling for this case. It will not emit an event for the newly-incoming touchpoint until after its matchGesture call has completed and determined which set of gestures the input may match. Said process relates strongly to the serialization scheme documented below and is the point where these two aspects interact. The event raised by onNewTrackedPath - 'inputstart' - should only fire once gesture processing may proceed for the input.
Serialization at this level may be further complicated for cases where a gesture state was being maintained due to having pushed an engine alt-state that had at least one live gesture; should the last sustaining sub-gesture reach completion by the new input, the sustain state must be unwound before continuing. For example, suppose a new menu appears when a key or button is held down, with this action triggering an "alt-mode" using a different base set of gesture models for use of that new menu while it displays. That base gesture may specify that it is to be kept alive to avoid interrupting interactions with the new menu when the "held down" key is released, only dropping the menu (and the "alt-mode") after those interactions resolve.
Touchpath sampling serialization
Delaying new inputs also invokes a need to delay events that occur at or after the same timestamp as the first delayed input. After all, tracking for any new input, including for the first delayed input, needs to be initiated before any future observations for such an input may be properly handled. This is especially critical for touch processing, as it is easily possible for the user to perform touch input with multiple fingers in parallel.
To address this, the touch-event processing component - TouchEventEngine - captures events as they come in and adds them to a processing queue, implemented as AsyncClosureDispatchQueue, that ensures the events stay in proper sequential, serialized order by use of Promises. It also creates specialized Promise-based locks whenever a new touchpoint is received that are added to the serialization queue; these locks are only released when TouchpointCoordinator.onNewTrackedPath has proceeded far enough to publish the touchpoint's tracking GestureSource instance - this is the purpose of the fulfillInputStart call within onNewTrackedPath.
These locks are important in case external code wishes to receive each incoming input event for the touchpoint via event; path event listeners can't be registered without access to the base touchpath object.
Use of the AsyncClosureDispatchQueue comes with an added benefit; it enforces a macrotask queue wait after gesture-level processing completes for each input event. This gives newly-incoming input events a chance to be recorded in a responsive fashion, which is particularly important for any gestures that require a sense of timing. This way, the true, original time of the input events will stay reasonably accurate, even if the actual gesture-level processing operations are delayed until later.