spiegel-keyman/web/testing/regression-tests/README.md

# Regression Tests

This folder contains tools to run automated regression tests on Keyman Engine for Web 
and Keyman Developer. The intent is to validate changes to the compiler and the engine
against the set of available keyboards in the Keyman keyboards repository from
https://github.com/keymanapp/keyboards.

Currently, the regression tests run only on Windows because we have a dependency on
the `kmanalyze` program which is not yet available outside the keyman source repo. This
is the only Windows dependency (the `kmcomp` compiler is also used, but this runs under
WINE and is available in binary form outside the source repo).

The test suite runs in Node.js and launches Chrome for the tests. In the future, the 
tests may be able to be run in native Node.js.

## Configuration

The test suite expects the keyboards repository to be located as a sibling folder of
the keyman repository, for example:

    C:\Projects\keyman
    - keyboards (keyboards.git)
      - release
        - k
          - khmer_angkor
      - ...
    - apps (keyman.git)
      - windows
        - src
          - ...
      - web
        - src
          - ...
        - testing
          - regression-tests

Keyman Engine for Web, Keyman Developer (kmcomp, kmcmpdll, kmanalyze) must be built
prior to running the tests. (If not testing source versions, only kmanalyze is 
required). See the build documentation for each project for details; summary below:

To build KeymanWeb (bash):

    cd web/source
    ./build.sh

To build just the Keyman Developer tools required from the Windows source (cmd):

    cd windows\src
    make -DNOUI ext global buildtools developer

### Known Failures

As there are some known failures, these are listed in `src/known-failures.js`. 
As the bugs that impact these keyboards are addressed, they should be progressively
removed from the file. Keyboards listed in this file will still be tested and
report errors, but they will not fail the test suite overall.

## Running Tests

An interactive or manual test web page is available, as well as the automated
tests that run with node + Karma + Mocha + Chai.

### Manual Tests

`node interactive.js` will start a web server listening on http://localhost:1337/. 
Navigating to this address will allow you to load and test a given keyboard from the
keyboards repository. This assumes that the keyboards are already available and that
the .tests have already been built.

This is designed more for validating the testing framework and changes to the testing
tools than for intensive testing.

### Automated Tests

`node test.js` runs the entire test suite. There are a number of command line 
parameters available:

    $ node test.js -h
    Usage: test [options]

    Options:
      -V, --version                       output the version number
      -c, --compiler-versions [versions]  Specify compiler version(s) to test. Can specify "stable", "source" or a specific version number. (default: ["stable","source"])
      -e, --engine-versions [versions]    Specify KeymanWeb engine version(s) to test. Can specify "stable", "source" or a specific version number. (default: ["stable","source"])
      -k, --keyboards [keyboards]         Builds and tests specific keyboard source files. If -k is not specified, then test all keyboards in the keyboards repository. (default: [])
      -f, --fail-fast                     Don't attempt to continue tests after the first keyboard test fails
      --deep                              Compare all version combinations against base version, instead of just one; only valid when comparing 1 version of each against base
      --skip-analysis                     Don't create .tests files, assume they are already present
      -l, --log-all-failures              Log all test failures to console, not just the first failure for each keyboard
      -h, --help                          output usage information

When the entire test suite is run, the regression test compares the latest version of 
the compiler from the source repository, together with the latest version of Keyman
Engine for Web from the source repository, against the latest stable releases of 
each, downloaded from https://downloads.keyman.com/

These two versions are referenced as `source` and `stable` respectively. You can
also specify exact stable, beta or alpha versions for download by specifying a
version number in the `-c` and `-e` parameters. You can test as many versions as
you like at a time, but beware: the number of permutations tested increases as 
the square of tested versions.

The continuous integration configuration currently uses `node test.js -l`.

By default, the tests will run through all keyboards, even after a failure, except
for catastrophic errors (such as timeouts, missing test files).

#### Quick run of tests

`node test-builder.js` can be used to re-run a specific test without downloading
tools or recompiling the keyboard. Full command line:

    $ node test-builder.js -h
    Usage: test-builder [options]

    Options:
      -V, --version                     output the version number
      -c, --compiler-version [version]  Specify compiler version that is being tested, which is saved in the results file.
      -e, --engine-version [version]    Specify KeymanWeb engine version that is being tested, which is saved in the results file.
      -k, --keyboards [keyboards]       Test specific keyboard(s) in the repo, e.g. k/khmer_angkor. If -k is not specified,
                                        then test all keyboards in the keyboards repository. (default: [])
      -h, --help                        output usage information

The test builder generates a `tests-generated.js` file for running Karma and then launches Karma with it.

## Background

`kmanalyze` generates a .tests file for a given keyboard, from analysis of a
corresponding .kmx file (a future update will use the intermediate in-memory 
.kmx file used by the compiler to ensure we access only the web rules). It
does not currently support deadkeys, recursive groups, or web-specific rules
(lines prefixed with `$keymanweb:`). Nor does it support keyboard or system
options. It will generate a single test for each store referenced in an `any`
statement, rather than a test for each character in the store.

.tests files and .results files are stored in a `tests/` folder for each 
keyboard in the repository.

The test suite runs tests against the physical keyboard ruleset for keyboards.
`src/test-runner.js` contains the code for running the individual tests. These
could be extended in future to support touch tests as well as other functionality,
e.g. by parameterising the `inputTests` property of the .tests file.