diff --git a/docs/linux-packaging.md b/docs/linux-packaging.md new file mode 100644 index 0000000000..c71f4f08e6 --- /dev/null +++ b/docs/linux-packaging.md @@ -0,0 +1,258 @@ +# Linux packaging + +## Package distribution + +We use different channels to build and distribute the Linux packages: + +- (older version) included in official repos since Ubuntu 19.04 and Debian Buster +- Launchpad repo for [stable](https://launchpad.net/~keymanapp/+archive/ubuntu/keyman) and + [beta](https://launchpad.net/~keymanapp/+archive/ubuntu/keyman-daily) versions +- [pso](http://packages.sil.org/) and [llso](http://linux.lsdev.sil.org/ubuntu/) + for stable, beta, and alpha versions +- artifacts on [Jenkins](https://jenkins.lsdev.sil.org/view/Keyman/view/Pipeline/job/pipeline-keyman-packaging/view/change-requests/) + for pull requests + +Packages on [llso](http://linux.lsdev.sil.org/ubuntu/) are uploaded automatically and are +intended as a staging environment for testing. Packages on [pso](http://packages.sil.org/) +are uploaded manually after the packages received some testing. End users usually have only +pso enabled. + +## Package builds + +Package builds happen on Launchpad and Jenkins. Package builds for the official Ubuntu/Debian +repos happen outside of our control. + +## Package builds on Jenkins + +### Build jobs + +The definition of the packaging jobs, the triggering of the jobs and the necessary build scripts +are scattered over several source repos: + +- [ci-builder-scripts](https://github.com/sillsdev/ci-builder-scripts) contains the definition of + a meta job (multi-branch pipeline job) that gets triggered when a change gets pushed to the + [Keyman GitHub repo](https://github.com/keymanapp/keyman). The meta job creates a new build + configuration/job for each branch/pull request on GitHub. The new job gets triggered to initialize + itself, but then exits immediately. We use the Jenkins + [Job DSL plugin](https://github.com/jenkinsci/job-dsl-plugin/wiki) to define the meta job. + + ci-builder-scripts also contains several generic scripts to set up a package build environment + (using `sbuilder`) and for building source and binary packages. These scripts are shared with + other projects. + + The Keyman GitHub repo defines a webhook that triggers the meta job on Jenkins. + + Changes to ci-builder-scripts go through [Gerrit](https://gerrit.lsdev.sil.org). See + [CONTRIBUTING.md](https://github.com/sillsdev/ci-builder-scripts/blob/master/CONTRIBUTING.md) + for details. + + File structure: + + - [groovy/KeymanPackagingJobs.groovy](https://github.com/sillsdev/ci-builder-scripts/blob/master/groovy/KeymanPackagingJobs.groovy) + contains the meta job definition + - The [bash/](https://github.com/sillsdev/ci-builder-scripts/tree/master/bash) subdirectory + contains `bash` scripts: + + - [setup.sh](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/setup.sh) - + setup sbuild chroot environment + - [update](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/update) - + update the sbuild chroot environment + - [build-package](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/build-package) - + create a binary package + +- [lsdev-pipeline-library](https://github.com/sillsdev/lsdev-pipeline-library) contains a reusable + Jenkins pipeline library. The + [vars/keymanPackaging.groovy](https://github.com/sillsdev/lsdev-pipeline-library/blob/master/vars/keymanPackaging.groovy) + file contains the bulk of the logic of the Keyman packaging job. + +- The [Keyman GitHub repo](https://github.com/keymanapp/keyman) contains various scripts that are + used to trigger a build and as part of the package build, and of course the source code for the + packages: + + - [resources/build/run-required-test-builds.sh](https://github.com/keymanapp/keyman/blob/master/resources/build/run-required-test-builds.sh) + runs on [TeamCity](https://build.palaso.org/buildConfiguration/Keyman_Test?) to trigger the + builds for the various platforms, among them the Jenkins package build. + - [resources/build/increment-version.sh](https://github.com/keymanapp/keyman/blob/master/resources/build/increment-version.sh) + runs on [TeamCity](https://build.palaso.org/buildConfiguration/Keyman_TriggerReleaseBuildsMaster?) + and increments the version number before triggering the builds for the various platforms. + - [linux/Jenkinsfile](https://github.com/keymanapp/keyman/blob/master/linux/Jenkinsfile) is a flag + for the meta job. If the meta job finds this file, it will create a new build configuration. This + file simply calls the packaging functionality defined in `lsdev-pipeline-library` and passes the + distributions and architectures to build as parameters. + - [linux/build/agent/install-deps](https://github.com/keymanapp/keyman/blob/master/linux/build/agent/install-deps) + installs dependencies on the current build agent. + - The [linux/scripts](https://github.com/keymanapp/keyman/tree/master/linux/scripts) subdirectory + contains `bash` scripts that are used during the package build. Some are only needed for + Launchpad builds. + + - [jenkins.sh](https://github.com/keymanapp/keyman/blob/master/linux/scripts/jenkins.sh) + gets called from `lsdev-pipeline-library` to create a source package. + + - [linux/*/debian](https://github.com/keymanapp/keyman/tree/master/linux/ibus-kmfl/debian) - + each package has a separate `debian` subdirectory with the meta data for the Linux package. + See [Debian New Maintainers' Guide](https://www.debian.org/doc/manuals/maint-guide/) for + details to the various files. + +### Flow of a Linux package build + +- [TeamCity job](https://build.palaso.org/buildConfiguration/Keyman_Test?) triggers a build on + [Jenkins](https://jenkins.lsdev.sil.org/view/Keyman/view/Pipeline/job/pipeline-keyman-packaging/) +- Jenkins verifies the build parameters and starts the matching build configuration for the PR or + branch +- The [build job](https://github.com/sillsdev/lsdev-pipeline-library/blob/master/vars/keymanPackaging.groovy) runs several checks: + + - it exits immediately if the build is not manually triggered and no parameters are passed in + (i.e. it got triggered by the GitHub webhook) + - it doesn't build if this is a PR, didn't get triggered manually and the PR is not from a trusted + user + - it doesn't build if no Linux-relevant files changed unless the parameter `force` was passed + - manually triggered builds will always build + +- build job installs + [dependencies](https://github.com/keymanapp/keyman/blob/master/linux/build/agent/install-deps) + on the current build agent +- build job creates a source package for the linux packages (keyman-keyboardprocessor, kmflcomp, + libkmfl, ibus-kmfl, keyman-config, and ibus-keyman). This is done by calling + [scripts/jenkins.sh](https://github.com/keymanapp/keyman/blob/master/linux/scripts/jenkins.sh). +- build job creates the binary package for each linux package on each distribution (currently + xenial, bionic) and each architecture (amd64, i386) +- at the end of the build if it is not a build of a PR, the `.deb` file gets uploaded to llso + (alpha packages to e.g. `bionic-experimental`, beta packages to `bionic-proposed` and + packages build from the stable branch to the main section `bionic`) +- if the build is successful the job archives the artifacts + +The Jenkins build progress is visible in two ways: + +- [traditional view](https://jenkins.lsdev.sil.org/view/Keyman/view/Pipeline/job/pipeline-keyman-packaging/) +- [blue ocean view](https://jenkins.lsdev.sil.org/blue/organizations/jenkins/pipeline-keyman-packaging/activity) + +### Local package builds + +It is possible to use the usual Debian/Ubuntu tools to create the package locally. For someone who +only occasionally deals with packaging it might be easier to use the scripts that Jenkins runs: + +#### Prerequisites for local package builds + +Install `sbuild` (and probably some other packages that I forgot). + +You’ll need a chroot image before you can use sbuild. The scripts in +[ci-builder-scripts](https://github.com/sillsdev/ci-builder-scripts) will help +with that. [`setup.sh`](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/setup.sh) +can setup such chroots: + +```bash +bash/setup.sh --dists "xenial bionic" --arches "amd64 i386" +``` + +[`update`](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/update) is used to +later update those chroots: + +```bash +bash/update --dists "xenial bionic" --arches "amd64 i386" +``` + +Set the `DEBSIGNKEY` environment variable to your public GPG key that will be used to sign +the packages. + +#### Building packages + +Building packages happen in the [Keyman source tree](https://github.com/keymanapp/keyman). + +The Keyman +[`linux/scripts/jenkins.sh`](https://github.com/keymanapp/keyman/blob/master/linux/scripts/jenkins.sh) +script can be used to create a source package (replace `packageName` with the name of the package, +i.e. one of keyman-keyboardprocessor, kmflcomp, libkmfl, ibus-kmfl, keyman-config, and ibus-keyman). + +```bash +cd linux +./scripts/jenkins.sh ${packageName} ${DEBSIGNKEY} +``` + +This creates a source package (`_-1.dsc`) and some `*.tar.?z` files in the +`linux/` subdirectory. + +**NOTE:** The subdirectory for `keyman-keyboardprocessor` is `common/engine/keyboardprocessor`, for +all other packages `linux/`. + +ci-builder-script's [`build-package`](https://github.com/sillsdev/ci-builder-scripts/blob/master/bash/build-package) +script creates the binary packages: + +```bash +cd linux/${packageName} +~/ci-builder-scripts/bash/build-package \ + --dists "xenial bionic" --arches "amd64 i386" \ + --main-package-name "My great package" \ + --supported-distros "xenial bionic focal" \ + --debkeyid ${DEBSIGNKEY} --build-in-place --no-upload +``` + +This will create the binary package `_-1+1_.deb`. + +To speed up package building you might want to limit the build to a single dist +(e.g. `--dists "bionic"`) and arch (e.g. `--arches "amd64"`). + +After building packages it might be a good idea to clean up the source tree before doing further +work: + +```bash +git clean -dxf +``` + +## Package builds on Launchpad + +Package builds on Launchpad are triggered manually by running the Keyman script +[linux/scripts/launchpad.sh](https://github.com/keymanapp/keyman/blob/master/linux/scripts/launchpad.sh). + +### Prerequisites for Launchpad + +1. If you don't have one, create an account at [launchpad.net](https://launchpad.net) +2. Request to join the ["Keyman for Linux"](https://launchpad.net/~keymanapp) team. +3. Create a [GPG](https://help.ubuntu.com/community/GnuPrivacyGuardHowto) key and associate it + to your launchpad account +4. Set the following environment variables in your `~/.profile` or `~/.bashrc` (so you don't have + to set them every time) + `export GPGKEY=[key_id]` using the `key_id` of your GPG key + `DEBEMAIL="your.email.address@example.org"` + `DEBFULLNAME="Firstname Lastname"` + `export DEBEMAIL DEBFULLNAME` + +### Building packages on Launchpad + +The `launchpad.sh` script downloads the current source code (beta or stable) from +[downloads.keyman.com](https://downloads.keyman.com/linux/stable/), creates a Debian source package +and uploads this to launchpad. Launchpad then rebuilds for the different distros and architectures. + +To upload the packages to launchpad, run the following script from the `linux/` directory: + +```bash +./scripts/launchpad.sh [UPLOAD="yes"] [TIER=""] [PROJECT=""] [DIST=""] [PACKAGEVERSION=""] +``` + +#### Parameters + +- `UPLOAD="yes"` - do the dput for real +- `TIER=""` - alpha, beta, or stable, default from `../TIER.md` +- `PROJECT=""` - only upload this package +- `DIST=""` - only upload for this distribution +- `PACKAGEVERSION=""` - normally use the default so don't specify it. But if you + change packaging and run another upload you need to increment the number at the end of + `PACKAGEVERSION`. e.g. next one is `1~sil2` then `1~sil3`… + +### Releasing a new version + +As part of releasing a new version it might be good to do some local testing first before uploading +to Launchpad: + +- Run `launchpad.sh` with `UPLOAD="no"` to build the packages +- Then install them on a clean VM and make sure no glaring bugs +- Once you are happy with the packages you can run `launchpad.sh` with `UPLOAD="yes"` + +### Troubleshooting + +Refer to the [launchpad uploading help](https://help.launchpad.net/Packaging/PPA/Uploading) +for troubleshooting and setting up for `dput` upload. + +## Reference + +See the [Linux readme](https://github.com/keymanapp/keyman/blob/master/linux/README.md) +for how to build Keyman on Linux etc. diff --git a/linux/README.md b/linux/README.md index 9d93449947..2660b51579 100644 --- a/linux/README.md +++ b/linux/README.md @@ -29,7 +29,7 @@ * `make fullbuild` to configure and build * `sudo make install` to install to /usr/local -- Some of the files must be installed to `/usr/share/` so `make install` must be run as `sudo`. +- Some of the files must be installed to `/usr/share/` so `make install` must be run as `sudo`. - To do this run ` sudo make install` @@ -41,7 +41,7 @@ * If you already have the ibus-keyman package installed then it will move the file `/usr/share/ibus/component/keyman.xml` to `/usr/share/doc/ibus-keyman/` * run `sudo make uninstall` to uninstall everything and put it back again - + ##### Tmp install Used by TC for validating PRs @@ -54,22 +54,22 @@ This is only for testing the build, not for running ibus-kmfl or ibus-keyman in * libkmfl requires kmfl.h header from kmflcomp * ibus-kfml requires kmfl.h and headers and lib from libkmfl * ibus-keyman requires headers and lib from keyboardprocessor - - So + + So * kmflcomp must be built and installed before libkmfl * libkmfl must be built and installed before ibus-kmfl * keyboardprocessor must be built before ibus-keyman - + For each project run `./configure && make && make install`. - - You may prefer to create a different directory to build in and run configure from there e.g. - + + You may prefer to create a different directory to build in and run configure from there e.g. + `mkdir ../build-kmflcomp;` `cd ../build-kmflcomp` - `../kmflcomp/configure` + `../kmflcomp/configure` `make ` `make install` - + The install of ibus-kmfl doesn't install everything to the correct location for it to be used - to be fixed ### Continuous integration @@ -80,35 +80,11 @@ Nightly builds upload the most recent new master build to https://downloads.keym ### Building packages -Jenkins now continuously builds Debian packages on every commit to master -Periodically test packages will be uploaded to https://launchpad.net/~keymanapp/+archive/ubuntu/keyman-daily - -### Launchpad -This section is primarily for "Keyman for Linux" team members, and not needed for general building. - -1. If you don't have one, create an account at https://launchpad.net -2. Request to join the ["Keyman for Linux"](https://launchpad.net/~keymanapp) team. -3. Create a [GPG](https://help.ubuntu.com/community/GnuPrivacyGuardHowto) key and associate it to your launchpad account -4. Set the following environment variables in your ~/.profile or ~/.bashrc (so you don't have to set them every time) - `export GPGKEY=[key_id]` using the `key_id` of your GPG key - `DEBEMAIL="your.email.address@example.org"` - `DEBFULLNAME="Firstname Lastname"` - `export DEBEMAIL DEBFULLNAME` - -To upload the packages to launchpad, run the following script from the `linux/` directory: -``` -./scripts/launchpad.sh [UPLOAD="yes"] [TIER=""] [PROJECT=""] [DIST=""] [PACKAGEVERSION=""] -``` -**Parameters** -UPLOAD="yes" do the dput for real -TIER="\" alpha, beta, or stable, default from ../TIER.md -PROJECT="\" only upload this project -DIST="\" only upload for this distribution -PACKAGEVERSION="\" normally use the default so don't specify it. But if you change packaging and run another upload you need to increment the number at the end of `PACKAGEVERSION`. e.g. next one is `1~sil2` then `1~sil3`... - -Refer to https://help.launchpad.net/Packaging/PPA/Uploading for troubleshooting and setting up for dput upload. +See [Linux packaging doc](https://github.com/keymanapp/keyman/blob/master/docs/linux-packaging.md) +for details on building Linux packages for Keyman. ### Testing + Tests to be created as there are no current tests ### Running Keyman for Linux @@ -164,4 +140,4 @@ Use `Win-space` to switch between keyboards. * Click `Add` to add a keyman keyboard. * Click the 3 dots expander then search for "Other" and click it * The Keyman keyboards should be listed here to choose - +