spiegel-keyman/docs/linux-packaging.md

Linux packaging

Package distribution

We use different channels to build and distribute the Linux packages:

• (older version) included in official repos since Ubuntu 19.04 and Debian Buster
• Launchpad repo for stable and beta versions
• pso and llso for stable, beta, and alpha versions
• artifacts on Jenkins for pull requests

Packages on llso are uploaded automatically and are intended as a staging environment for testing. Packages on pso are uploaded manually after the packages received some testing. End users usually have only pso enabled.

Package builds

Package builds happen on Launchpad and Jenkins. Package builds for the official Ubuntu/Debian repos happen outside of our control.

Package builds on Jenkins

Build jobs

The definition of the packaging jobs, the triggering of the jobs and the necessary build scripts are scattered over several source repos:

• ci-builder-scripts contains the definition of a meta job (multi-branch pipeline job) that gets triggered when a change gets pushed to the Keyman GitHub repo. The meta job creates a new build configuration/job for each branch/pull request on GitHub. The new job gets triggered to initialize itself, but then exits immediately. We use the Jenkins Job DSL plugin to define the meta job.

  ci-builder-scripts also contains several generic scripts to set up a package build environment (using sbuilder) and for building source and binary packages. These scripts are shared with other projects.

  The Keyman GitHub repo defines a webhook that triggers the meta job on Jenkins.

  Changes to ci-builder-scripts go through Gerrit. See CONTRIBUTING.md for details.

  File structure:

  • groovy/KeymanPackagingJobs.groovy contains the meta job definition

  • The bash/ subdirectory contains bash scripts:

    • setup.sh - setup sbuild chroot environment
    • update - update the sbuild chroot environment
    • build-package - create a binary package

• lsdev-pipeline-library contains a reusable Jenkins pipeline library. The vars/keymanPackaging.groovy file contains the bulk of the logic of the Keyman packaging job.

• The Keyman GitHub repo contains various scripts that are used to trigger a build and as part of the package build, and of course the source code for the packages:

  • resources/build/run-required-test-builds.sh runs on TeamCity to trigger the builds for the various platforms, among them the Jenkins package build.

  • resources/build/increment-version.sh runs on TeamCity and increments the version number before triggering the builds for the various platforms.

  • linux/Jenkinsfile is a flag for the meta job. If the meta job finds this file, it will create a new build configuration. This file simply calls the packaging functionality defined in lsdev-pipeline-library and passes the distributions and architectures to build as parameters.

  • linux/build/agent/install-deps installs dependencies on the current build agent.

  • The linux/scripts subdirectory contains bash scripts that are used during the package build. Some are only needed for Launchpad builds.

    • jenkins.sh gets called from lsdev-pipeline-library to create a source package.

  • linux/*/debian - each package has a separate debian subdirectory with the meta data for the Linux package. See Debian New Maintainers' Guide for details to the various files.

Flow of a Linux package build

• TeamCity job triggers a build on Jenkins

• Jenkins verifies the build parameters and starts the matching build configuration for the PR or branch

• The build job runs several checks:

  • it exits immediately if the build is not manually triggered and no parameters are passed in (i.e. it got triggered by the GitHub webhook)
  • it doesn't build if this is a PR, didn't get triggered manually and the PR is not from a trusted user
  • it doesn't build if no Linux-relevant files changed unless the parameter force was passed
  • manually triggered builds will always build

• build job installs dependencies on the current build agent

• build job creates a source package for the linux packages (keyman-keyboardprocessor, kmflcomp, libkmfl, ibus-kmfl, keyman-config, and ibus-keyman). This is done by calling scripts/jenkins.sh.

• build job creates the binary package for each linux package on each distribution (currently xenial, bionic) and each architecture (amd64, i386)

• at the end of the build if it is not a build of a PR, the .deb file gets uploaded to llso (alpha packages to e.g. bionic-experimental, beta packages to bionic-proposed and packages build from the stable branch to the main section bionic)

• if the build is successful the job archives the artifacts

The Jenkins build progress is visible in two ways:

• traditional view
• blue ocean view

Local package builds

It is possible to use the usual Debian/Ubuntu tools to create the package locally. For someone who only occasionally deals with packaging it might be easier to use the scripts that Jenkins runs:

Prerequisites for local package builds

Install sbuild (and probably some other packages that I forgot).

You’ll need a chroot image before you can use sbuild. The scripts in ci-builder-scripts will help with that. setup.sh can setup such chroots:

bash/setup.sh --dists "xenial bionic" --arches "amd64 i386"

update is used to later update those chroots:

bash/update --dists "xenial bionic" --arches "amd64 i386"

Set the DEBSIGNKEY environment variable to your public GPG key that will be used to sign the packages.

Building packages

Building packages happen in the Keyman source tree.

The Keyman linux/scripts/jenkins.sh script can be used to create a source package (replace packageName with the name of the package, i.e. one of keyman-keyboardprocessor, kmflcomp, libkmfl, ibus-kmfl, keyman-config, and ibus-keyman).

cd linux
./scripts/jenkins.sh ${packageName} ${DEBSIGNKEY}

This creates a source package (<packageName>_<version>-1.dsc) and some *.tar.?z files in the linux/<packageName> subdirectory.

NOTE: The subdirectory for keyman-keyboardprocessor is common/engine/keyboardprocessor, for all other packages linux/<packageName>.

ci-builder-script's build-package script creates the binary packages:

cd linux/${packageName}
~/ci-builder-scripts/bash/build-package \
    --dists "xenial bionic" --arches "amd64 i386" \
    --main-package-name "My great package" \
    --supported-distros "xenial bionic focal" \
    --debkeyid ${DEBSIGNKEY} --build-in-place --no-upload

This will create the binary package <packageName>_<version>-1+<dist>1_<arch>.deb.

To speed up package building you might want to limit the build to a single dist (e.g. --dists "bionic") and arch (e.g. --arches "amd64").

After building packages it might be a good idea to clean up the source tree before doing further work:

git clean -dxf

Package builds on Launchpad

Package builds on Launchpad are triggered manually by running the Keyman script linux/scripts/launchpad.sh.

Prerequisites for Launchpad

1. If you don't have one, create an account at launchpad.net
2. Request to join the "Keyman for Linux" team.
3. Create a GPG key and associate it to your launchpad account
4. Set the following environment variables in your ~/.profile or ~/.bashrc (so you don't have to set them every time) export GPGKEY=[key_id] using the key_id of your GPG key DEBEMAIL="your.email.address@example.org" DEBFULLNAME="Firstname Lastname" export DEBEMAIL DEBFULLNAME

Building packages on Launchpad

The launchpad.sh script downloads the current source code (beta or stable) from downloads.keyman.com, creates a Debian source package and uploads this to launchpad. Launchpad then rebuilds for the different distros and architectures.

To upload the packages to launchpad, run the following script from the linux/ directory:

./scripts/launchpad.sh [UPLOAD="yes"] [TIER="<tier>"] [PROJECT="<project>"] [DIST="<dist>"] [PACKAGEVERSION="<version>"]

Parameters

• UPLOAD="yes" - do the dput for real
• TIER="<tier>" - alpha, beta, or stable, default from ../TIER.md
• PROJECT="<project>" - only upload this package
• DIST="<dist>" - only upload for this distribution
• PACKAGEVERSION="<version>" - normally use the default so don't specify it. But if you change packaging and run another upload you need to increment the number at the end of PACKAGEVERSION. e.g. next one is 1~sil2 then 1~sil3…

Releasing a new version

As part of releasing a new version it might be good to do some local testing first before uploading to Launchpad:

• Run launchpad.sh with UPLOAD="no" to build the packages
• Then install them on a clean VM and make sure no glaring bugs
• Once you are happy with the packages you can run launchpad.sh with UPLOAD="yes"

Troubleshooting

Refer to the launchpad uploading help for troubleshooting and setting up for dput upload.

Reference

See the Linux readme for how to build Keyman on Linux etc.