spiegel-keyman/web/docs/internal/gestures/gesture-input-serialization.md

# Gesture Input Serialization

The GestureRecognizer engine is designed to handle and process asynchronous, continuous input of one or more touchpoints in parallel.  The processing of events takes time, though, and on older devices it is possible for new input events to come in while older events are still being processed.  As it is possible for an earlier event to affect how a later event should be processed, it is necessary to ensure that all events are processed in a specific order. We wish to capture the user's original intent for each aspect of input and later match it against the system state that a user would reasonably anticipate for the input based on their previous interactions, even if such a state hasn't yet actually been reached or activated at the true time of input.

For a keyboard, suppose a user taps the `Shift` and `A` keys with minimal time gap between the two events.  Also suppose that external code sets `state = 'shift'` when processing the `Shift` key, with `default` its current value - perhaps reflecting the name of different layers in an on-screen keyboard.  For this example, the `Shift` state may not be in place yet once the `A` is tapped - event processing does take _some_ time, after all - especially if the events are considered simultaneous by the browser.

Note that JS mouse and touch events lock in their target elements at the time of observation, even if that element is swapped out of the DOM by the time the event is actually handled.  The serialization scheme used by the engine handles the `A`s event (provisionally selecting the unshifted `A`, per the original `default` state) but delays further processing for it until the `Shift` state has taken effect.  If supplied with a properly-designed `itemIdentifier` method as part of the engine's configuration, the serialization processes then uses that method to finalize the associated element once processing resumes, remapping from the unshifted `A` to the Shift-layer `A` per the shift in the event's associated engine `state`.

Furthermore, the engine leverages `Promise`s internally to help keep the different layers of [gesture processing](./gesture-processing.md) separate and isolated; these need a chance to resolve before continuing processing of the new event and any events that may follow it.  In particular, cases like the `Shift` + `A` example invoke `setTimeout` waits that leverage the JS task queue, not just the microtask queue that `Promise`s natively leverage.  During times of rapid, heavy input, it is thus possible for newly-incoming touch events to be received before all `Promise`s for prior input gesture processing have fully resolved.  It is thus necessary to artificially defer processing of newly-incoming touch events so that their processing does not occur before cases like `Shift` + `A` above fully resolve.

## Gesture Processing serialization

The first point that requires special handling is detection of newly-received inputs - new mouse or touch interaction points.  As some gesture models may specify autocompletion whenever a new touch comes in, those should resolve before further handling the new input.

Toward this end, `MatcherSelector`'s `matchGesture` method maintains a `Promise` serving as a lock within the field `pendingMatchSetup`.  This is set whenever checking such cases - whether or not existing gestures have special interactions triggered by new touchpoints - and is cleared once these checks have completed.  This lock is referenced early within the same method to prevent `matchGesture` from handling any subsequent inputs while waiting on this process to complete.

The method calling this - `TouchpointCoordinator.onNewTrackedPath` - also includes special handling for this case.  It will not emit an event for the newly-incoming touchpoint until after its `matchGesture` call has completed and determined which set of gestures the input may match.  Said process relates strongly to the [serialization](#touchpath-sampling-serialization) scheme documented below and is the point where these two aspects interact.  The event raised by `onNewTrackedPath` - `'inputstart'` - should only fire once gesture processing may proceed for the input.

Serialization at this level may be further complicated for cases where a gesture state was being maintained due to having pushed an engine alt-state that had at least one live gesture; should the last sustaining sub-gesture reach completion by the new input, the sustain state must be unwound before continuing.  For example, suppose a new menu appears when a key or button is held down, with this action triggering an "alt-mode" using a different base set of gesture models for use of that new menu while it displays.  That base gesture may specify that it is to be kept alive to avoid interrupting interactions with the new menu when the "held down" key is released, only dropping the menu (and the "alt-mode") after those interactions resolve.

## Touchpath sampling serialization

Delaying new inputs also invokes a need to delay events that occur at or after the same timestamp as the first delayed input.  After all, tracking for any new input, including for the first delayed input, needs to be initiated _before_ any future observations for such an input may be properly handled.  This is especially critical for touch processing, as it is easily possible for the user to perform touch input with multiple fingers in parallel.

To address this, the touch-event processing component - `TouchEventEngine` - captures events as they come in and adds them to a processing queue, implemented as `AsyncClosureDispatchQueue`, that ensures the events stay in proper sequential, serialized order by use of `Promise`s.  It also creates specialized `Promise`-based locks whenever a new touchpoint is received that are added to the serialization queue; these locks are only released when `TouchpointCoordinator.onNewTrackedPath` has proceeded far enough to publish the touchpoint's tracking `GestureSource` instance - this is the purpose of the `fulfillInputStart` call within `onNewTrackedPath`.

These locks are important in case external code wishes to receive each incoming input event for the touchpoint via event; path event listeners can't be registered without access to the base touchpath object.

Use of the `AsyncClosureDispatchQueue` comes with an added benefit; it enforces a macrotask queue wait after gesture-level processing completes for each input event.  This gives newly-incoming input events a chance to be recorded in a responsive fashion, which is particularly important for any gestures that require a sense of timing.  This way, the true, original time of the input events will stay reasonably accurate, even if the actual gesture-level processing operations are delayed until later.