Keyman/Language Modelling layer ================================ This document introduces the protocol for communicating between KeymanWeb and the language modeling layer (i.e., the configurable prediction and suggestion engine, (henceforth referred to as the "LMLayer"). Note: I'm using the term **keyboard** as a synonym for **KeymanWeb**. > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL > NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and > "OPTIONAL" in this document are to be interpreted as described in > [RFC 2119]. [RFC 2119]: https://www.ietf.org/rfc/rfc2119.txt Communication protocol between keyboard and asynchronous worker --------------------------------------------------------------- ![Sequence diagram of obtaining a prediction](./predictive-text-sequence.png) We have decided that everything to the right of the `KeymanWeb` will be in a Web Worker. However, communication can happen only through [`postMessage(data)`][postMessage] commands, where `data` is a serializable object (via the [structured clone][] algorithm). What serializable object can we send that will adhere to the [open-closed principle]? ### Messages The idea is to use a [discriminated union][]. The protocol involves plain JavaScript objects with one property called `message` that takes a finite set of `string` values. These string values indicate what message should be sent. The rest of the properties in the object are the parameters send with the message. ```javascript { message: 'predict', // message-specific properties here } ``` See also: [XML-RPC][] Messages are **not** methods. That is, there is no assumption that a client will receive a reply when a message is sent. However, some pairs of messages, such as `predict` and `suggestions`, assume (non-blocking!) request/response semantics. [discriminated union]: http://www.typescriptlang.org/docs/handbook/advanced-types.html#discriminated-unions [open-closed principle]: https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle [XML-RPC]: https://en.wikipedia.org/wiki/XML-RPC [structured clone]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm [postMessage]: https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage ### Tokens Tokens uniquely identify an input event, such as a key press. Since Keyman should ask for an asynchronous prediction on most key presses, the token is intended to associate a prediction and its response with a particular input; Keyman is free to ignore prediction responses if they are for outdated input events. The `Token` type is opaque to LMLayer. That is, LMLayer does not inspect its contents; it simply uses it to identify a request and pass it back to Keyman. There are a few requirements on the concrete type of the `Token`: 1. Tokens **MUST** be serializable via the [structured clone][] algorithm; 2. Tokens **MUST** be usable as a key in a [`Map`][Map object] object. 3. Tokens **MUST** be unique across messages. That is, tokens **MUST NOT** be duplicated between different messages. It is up to the keyboard to create unambiguous tokens that can be uniquely identified through the round-trip process. In the following examples, I'll use the subset of `number` values that are interpretable as [31-bit signed integers][Smi]. [Map object]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map#Key_equality [Smi]: https://github.com/thlorenz/v8-perf/blob/master/data-types.md#efficiently-representing-values-and-tagging [Tokens]: #tokens ### Example An asynchronous message to predict after typing 'D': ```javascript { message: 'predict', token: 1, transform: { insert: 'D', deleteLeft: 0, deleteRight: 0 }, context: { left: '', right: '', startOfBuffer: true, endOfBuffer: true }, } ``` Message types ------------- Currently there are four message types: Message | Direction | Parameters | Expected reply | Uses token --------------|--------------------|---------------------|---------------------|--------------- `initialize` | keyboard → LMLayer | capabilities, model | Yes — `ready` | No `ready` | LMLayer → keyboard | configuration | No | No `predict` | keyboard → LMLayer | transform, context | Yes — `suggestions` | Yes `suggestions` | LMLayer → keyboard | suggestions | No | Yes ### Message: `initialize` Must be sent from the keyboard to the LMLayer so that the LMLayer initializes a model. It will send `initialization` which is a plain JavaScript object specify the path to the model, as well configurations and platform restrictions. The keyboard **MUST NOT** send any messages to the LMLayer prior to sending `initialize`. The keyboard **SHOULD NOT** send another message to the keyboard until it receives `ready` message from the LMLayer before sending another message. The LMLayer needs to know the platform's abilities and restrictions (capabilities), as well as which concrete language model to instantiate. These properties are passed as `capabilities` and `model`, respectively. ```typescript interface InitializeMessage { message: 'initialize'; /** * A ModelDescription that describes the language model and its parameters. * The concrete documentation on is a valid ModelDescription * can be found elsewhere. */ model: { /** * What kind of model to instantiate. This is subject to availability, * but common examples are 'wordlist', 'fst', and 'dummy'. */ type: string; /** * Each model type defines a set of configurable parameters. Please * see the corresponding model's documentation for an extensive list. */ ...parameters: any; }; capabilities: { /** * Whether the platform supports deleting to the right. * The absence of this rule implies false. */ supportsDeleteRight?: false, /** * The maximum amount of UTF-16 code units that the keyboard will * provide to the left of the cursor, as an integer. */ maxLeftContextCodeUnits: number, /** * The maximum amount of code units that the keyboard will provide to * the right of the cursor, as an integer. The absence of this rule * implies the platform is incapable of supplying right contexts. * See also, [[supportsRightContexts]]. */ maxRightContextCodeUnits?: number, } } ``` ### Message: `ready` Must be sent from the LMLayer to the keyboard when the LMLayer's model as a response to `initialize`. It will send `configuration`, which is a plain JavaScript object requesting configuration from the keyboard. There are only two options defined so far: ```typescript interface ReadyMessage { message: 'ready'; configuration: { /** * How many UTF-16 code units maximum to send as the context to the * left of the cursor ("left" in the Unicode character stream). * * Affects the `context` property sent in `predict` messages. * * While the left context MUST NOT bisect surrogate pairs, they MAY * bisect graphical clusters. */ leftContextCodeUnits: number, /** * How many UTF-16 code units maximum to send as the context to the * right of the cursor ("right" in the Unicode character stream). * * Affects the `context` property sent in `predict` messages. * * While the left context MUST NOT bisect surrogate pairs, they MAY * bisect graphical clusters. */ rightContextCodeUnits: number, }; } ``` ### Message: `predict` Sent from the keyboard to the LMLayer whenever a new prediction should be generated. This is typically initiated by a key press event. The keyboard **SHOULD** track each `predict` message using a [token][Tokens]. The token **MUST** be unique across all prediction events. The LMLayer **SHOULD** respond to each `predict` message with a `suggestions` message. The `suggestions` message **MUST** contain the corresponding token as sent in the initial `predict` message. The keyboard **MUST** send the `context` parameter. The keyboard **SHOULD** send the `transform` parameter. The keyboard **MUST** send a unique token. The semantics of the `predict` message **MUST** be from the perspective of this sequence of events: 1. After the input event is received by the keyboard. 2. Before the keyboard applies the associated `transform` to the buffer. **NOTE**: The keyboard **MAY** apply the `transform` associated with the input event before receiving the corresponding `suggestions` message from the LMLayer. The intention is that once the suggestions are displayed, the typist may select one of the suggestions in the place of the effects of their original input. **NOTE**: The keyboard **MAY** send the `predict` message after it has applied the `transform` associated with the input event to the buffer. Regardless of the actual sequence, the semantics **MUST** remain the same: the prediction happens from the perspective before the `transform` has been applied. Therefore, if the keyboard eagerly transforms the buffer before it has sent the `predict` message, it must anticipate _undoing_ the `transform` it has already applied should it apply a `transfrom` sent in the `suggestions` message. The keyboard must act as if it has never applied the `transform` associated with the input event in the first place. The context is the text surrounding the insertion point, _before_ the transform is applied to the buffer. ```typescript interface Context { /** * Up to maxLeftContextCodeUnits code units of Unicode scalar value * (i. e., characters) to the left of the insertion point in the * buffer. If there is nothing to the left of the buffer, this returns * an empty string. */ left: USVString; /** * Up to maxRightContextCodeUnits code units of Unicode scalar value * (i. e., characters) to the right of the insertion point in the * buffer. If there is nothing to the right of the buffer, this returns * an empty string. */ right?: USVString; /** * Whether the insertion point is at the start of the buffer. */ startOfBuffer: boolean; /** * Whether the insertion point is at the end of the buffer. */ endOfBuffer: boolean; } ``` The transform parameter describes how the input event will change the buffer. ```typescript interface Transform { /** * The Unicode scalar values (i.e., characters) to be inserted at the * cursor position. * * Corresponds to `s` in com.keyman.KeyboardInterface.output. */ insert: USVString; /** * The number of code units to delete to the left of the cursor. * * Corresponds to `dn` in com.keyman.KeyboardInterface.output. */ deleteLeft: number; /** * The number of code units to delete to the right of the cursor. * Not available on all platforms. */ deleteRight?: number; } ``` ### Message: `suggestions` The `suggestions` message is sent from the LMLayer to the keyboard. This message sends a ranked array of suggestions, in descending order of probability (i.e., entry `0` is most likely, followed by entry `1`, etc.). This message **MUST** be in response to a `predict` message, and it **MUST** respond with the corresponding [token]. ```typescript /** * `suggestions` is an ordered array of suggestion objects. * Each suggestion is a transform bundled with a `displayAs` property. */ let suggestions = Suggestion[]; interface Suggestion { /** * Same object as an input event transform. * Note that the transform is applied AFTER the input event * transform. */ transform: Transform; /** * A string to display the suggestion to the typist. * This should aid the typist understand what the transform * will do to their text. */ displayAs: string; } ``` ```javascript let suggestions = [ { transform: { insert: 'teapot', deleteLeft: 1, deleteRight: 0 }, displayAs: '🍵' } ]; ``` #### Timing Each suggestion provides a `transform`. This transform is applied _after_ the transform associated with the input event that initiated this prediction. That is, the suggested transform applies to the buffer after the transform associated with the input event. Rephrased in somewhat mathematical terms: Let 𝑥 ∈ 𝐵 be the input text buffer. Let 𝑇𝑖 be the transform associated with an input event that maps a text buffer 𝐵 → 𝐵. Let 𝑇𝑠 be a transform suggested through the LMLayer. 𝑦 ∈ 𝐵 is the text buffer after the suggestion transform has been applied. The correct sequence of applications should be as follows: > 𝑦 = 𝑇𝑠(𝑇𝑖(𝑥)) #### Late suggestions Sometimes, a `suggestions` message may arrive after an input event has already invalidated its request. This is called a **"late" suggestion**. The LMLayer **MAY** send late suggestions. Consequently, the keyboard **MAY** discard the late suggestions. There is no requirement for the keyboard to acknowledge late suggestions, or for the LMLayer to avoid sending late `suggestions` messages. In either case, a `suggestions` message can be identified as appropriate or "late" via its `token` property.