After proceeding further in development, the only non-initFromReset initialization pattern that appears is constructing a raw instance, then immediately providing a computed tokenization. Thus, accepting the tokenization as an optional parameter will allow us a nice constructor spec for both init styles.
Removes a number of unused functions from the file, renames remaining
functions to match our builder script prefix patterns, removes unused
imports from various build scripts (e.g. Android).
Fixes: #14453
Consolidates the node-related script functions into node.inc.sh, as part
of cleaning up the build scripts and making them easier to maintain into
the future.
Fixes: #14447
Clarifies the confusing builder.inc.sh / build-utils.sh distinction by
giving the scripts more appropriate names. Most build scripts should use
builder-full.inc.sh; some helper scripts can use builder-basic.inc.sh.
Documented in resources/build/README.md.
Renames:
* resources/build/builder.inc.sh to resources/build/builder-full.inc.sh
* resources/build/build-utils.sh to resources/build/builder-basic.inc.sh
Other changes:
* Moves Android-specific functions out of builder-basic.inc.sh and into
android/build.sh.
* Renames functions in builder-basic.inc.sh
More functions may be moved from builder-basic.inc.sh into utils.inc.sh
or other scripts in the future.
Fixes: #14065
Build-bot: build all
Test-bot: skip
Adds documentation for various functions, removes unused functions (a
couple of simple, very lightly used functions were unDRYed; these could
go back the other way across all shell scripts if necessary). TODO items
noted for follow-up refactoring.
Fixes: #14275
Relates-to: #14269
Build-bot: build all
Test-bot: skip
Note that this will temporarily cause predictive-text to throw out older context states (that may be valid rewind targets) in favor of newer ones (that are known to be invalid rewind targets) due to the current context-state caching logic. That'll be resolved later, in an upcoming PR - and preferably, this one shouldn't merge without that one.
This PR's changes aim to meet our current needs for the tracking of context tokenization (word boundaries) across edits... all while moving much closer to the design specified within the Correction-Search + Context-Tracking design doc.
Also places the class within its own separate source file.
Thanks to a PR review that caught an accidental restart that hid one of the Transforms
Co-authored-by: Eberhard Beilharz <ermshiperete@users.noreply.github.com>
Following from #14364, this PR integrates the new method with the main predictive-text context-tracking code, significantly reworking the `attemptMatchContext` method in the process. While further refactoring of the latter method is planned, this step allows us to verify that the new methods integrate properly with the main codebase in their current form.
This also comes with the benefit of simplifying `attemptMatchContext` _significantly_ - large parts of its code were refactored into `attemptTokenizedAlignment`, and the new logic patterns are generally more straightforward to parse and understand.
Following from #14363, this method performs context alignment calculations that may be
used to match forms of the context before and after an edit by aligning their tokens and
validating any edits that may have occurred.
Note that no 'tracked context' states are manipulated or altered by this method - it
solely calculates the alignment deltas needed to align the two contexts. Other methods
may then take these values and determine the edits that occurred during the associated
context transition as needed.
Note that the `attemptTokenizedAlignment` method is not integrated into the main codebase
for the predictive-text worker at this time. That said, this method _does_ integrate
the `isSubstitutionAlignable` method introduced by #14363.
This adds one new method within the predictive-text worker space: isSubstitutionAlignable. The method is designed to report whether or not two words are "related enough" to consider as an appropriate word-level "substitution" when matching the incoming context against previously-seen contexts - a process useful for facilitating delayed reversions, among other things.
It is not yet integrated with the main body of worker code, however.
Some of the npm audit warnings left after #13897 require an updated `mocha` version to remedy. As we only really use `mocha` to do unit-testing, it should be safe to update if all of our existing tests pass with the new version.
Test-bot: skip
