Continuous Integration

Last modified 05 Aug 2022 10:31 +02:00
Table of Contents

CI Server

Jenkins: https://jenkins.evolveum.com/

On the Jenkins you can see the state of our builds.

The rest of this page is rather technical information, probably only useful for midPoint developers.

Build information

TODO

3rd party libraries with custom forks

Sometimes we need to fix or change a library that is used in midPoint. Prime example is our fork of Quartz library. Such dependencies have .e<number> suffix to their original version, like 2.3.2.e3. Sometimes the dot separator is accidentally omitted, but we should be consistent.

Problems with this solution as of early 2022 were:

  • Building of these forks is not covered by any process, only manual builds exist (can be fixed easily).

  • Branching strategy is unclear - it should perhaps somehow follow the midPoint needs? This can also be made more obvious by introducing CI builds where the branch will be explicit.

Shortly used custom Logback 1.2.3e1 (dot separator forgotten here) was removed from the project in midPoint version 4.6 in favor of main-line 1.2.11 which contains necessary fixes.

New Jenkins server

New Jenkins is our default now: https://jenkins.evolveum.com/

  • Our Jenkins uses Kubernetes plugin to run the builds.

  • MidPoint/Prism builds are made public using Matrix Authorization Strategy plugin.

    • Global Security is configured to use Project-based Matrix Authorization Strategy and Anonymous user has Read permission in Overall category. No permissions for views are necessary, the views containing visible tasks will be automatically visible as well.

    • The rest is job configuration where Read permission in Job category is added for Anonymous user. This does exactly what we are used to from our old Jenkins.

  • Build Job is stored in our private GitLab as XML file.

  • Build pipeline which actually describes how the build is executed can be either in the same private repository or in the repository that we want to build.

    • The latter has an advantage that the same Git repository is used in job XML and makes triggering the build after push more straightforward.

  • Install GitHub plugin to allow GitHub hooks after push.

    • All GitHub repositories must be set up with this Webhook by our GitHub administrator.

    • Enable the webhook trigger in Jenkins job. This can be done either by changing the XML in our private GitLab and then uploading it via API, or one can change it on the UI, download the config XML and merge the changes to Git sources. Both ways are described in the private repo.

    • Poll SCM is also a viable option if we don’t want to build after each push. Use our home timezone TZ=Europe/Bratislava on the first line of the specification to get the expected results.

  • There are two flavors of pipelines - declarative or scripted. Declarative is generally the preferred approach, but we’re using scripted style because when we started with the pipelines we had scripted examples, and we weren’t aware of the difference. Read this article about the main differences. In our case, we couldn’t use post construct for mail notifications and used try/finally over all the stages instead. We may reconsider this later, it really wasn’t a conscious/informed decision.

TODO:

  • Remove GitHub hooks for jenkins-new.

  • Do we want another conntest build for at least LTS version?

  • Simple mailing notifications work, sophisticated templates can be added later. (Eng?)

  • Running external DB for DB-specific tests (Oracle, old PG, SQL Server). DONE for native and generic PG and SQL Server, Oracle is troublesome yet. (?)

    • Oracle XE 21 must be used, older versions fail with "[WARNING] ORA-00821: Specified value of sga_target 1536M is too small, needs to be at least 2416M", probably the container sees too many processors: https://github.com/oracle/docker-images/issues/1522

    • Oracle non-XE DB requires auth for image pull (or we can build our own image), not sure about licenses.

    • Current pipeline commited in midpoint is WIP, currently the container script tries to run before Oracle is fully up.

  • How to debug failed build? (V) Can we leave pod/containers available for some limited time?

    • Alternatively, if tests fail, perhaps download test.log (or target) as artifact. This is current solution, all logs are archived in some pipelines, if the build is unstable.

    • Alternatively, add non-public debug build with more parameters, including maven command.

  • Add job for our Quartz, use the right branch (likely not master). (V)

  • After complete migration we can stop using -Dconfig and we can remove TestSqlRepositoryConfigurationFactory class completely. Oracle runs on old Jenkins at this moment, so not yet.

Open questions:

  • If we want to download only current Git tarball, we need to somehow replace what combo of git-commit-id-plugin and GitDescribe Maven plugin are doing now.

    • The time savings of fetching the tarball only are tremendous, e.g. 10 seconds vs 2-4 minutes.

    • git-commit-id-plugin requires .git to figure out the branch and commit ID (which also requires) working git binary. When using mvn+JDK image, avoid using slim versions that do NOT have git. Check the build output for Setting Git Describe: v4.6devel…​ string, if it says unknown, the git binary is missing.

    • This manifests as unknown value for Git describe line in the About page.

    • We also want to fetch changes since the last build (unless Jenkins does it automagically during the SCM check or what).

  • There seems to be no environment variable with URL of SCM (GIT) provided by Jenkins. This was debugged by printenv | sort command in the pipeline that was part of the repository. This means, that besides the job definition (XML or UI) itself, you have to repeat the GIT URL in the pipeline itself.

    • Similarly, there is no way to communicate what branch is going to be built.

    This is not a big deal for single-branch build (any reasonably complex build), but it can be annoying for simple builds (localization, samples…​) where a single build could handle all the supported branches.

  • Related to the previous point - Jenkins can do Git checkout by itself, if no git step is in the pipeline - but what is the difference? Also, why do we use git step when the documentation states that "the git step is a simplified shorthand for a subset of the more powerful checkout step"?

Maven considerations when using pipeline

Maven on Jenkins is run differently than your normal command line mvn and does a few things we need to cover by explicit options:

  • We don’t need to pollute logs with transport progress for downloads/uploads, add -ntp option. On the other hand, dropping -ntp sometimes may show strange download behavior, wrong repo used, etc.

  • Add -B to avoid accidental color codes in logs (may be unnecessary, but let’s be safe).

  • Add -Dmaven.test.failure.ignore for multi-module build if you want to run the build to the end. This is what Maven plugin for Jenkins normally does, but mvn does not. This is not necessary if tests are skipped, of course (e.g. -DskipTests).

Other random pipeline notes

  • Job parameters (e.g. VERBOSE) are accessible in the pipeline like script properties, i.e. just writing VERBOSE in the code or ${VERBOSE} in double-quote string (GString).

    • However, the parameter must be provided for such a pipeline, otherwise groovy.lang.MissingPropertyException: No such property: VERBOSE for class: groovy.lang.Binding is thrown.

    • Usage of parameter can be easily made option by accessing it via params map, i.e. params.VERBOSE in the code or ${params.VERBOSE} in a GString.

    • Default value can be easily provided using Elvis operator ?:, e.g. in GString: ${params.VERBOSE ?: '0'}.

      • But don’t use GString if the result is not just the part of the string, just use plain elvis returning string instead. That is, don’t use "${params.VERBOSE ?: '0'}" when plain params.VERBOSE ?: '0' is enough.

    • Params are mostly Strings, but there are ways how to initialize integer variable as well. This may require also some try/catch for number parse, so think twice whether it really is necessary.

    • If parameter is accessed on many places, it may be easier to extract it at the beginning of the pipeline script to avoid repeated default fallback:

      def verbose = params.VERBOSE ?: '0' // now use verbose variable lower instead params.VERBOSE

    • Note, that when ${verbose} is used inside GString it is replaced by the pipeline (Groovy) interpreter. If it is used in string (or multi-line string) defining shell script, it is already a resolved value for that script. Single-quote string (plain, non-G string) can be used and ${whatever} will be left for the shell script to resolve.

  • While not exactly hallmark of Reproducible build, it may be practical to have Maven+JDK container name parametrized. (Reproducibility can be saved by providing the default assumed container via Elvis operator.) This allows using the same pipeline for multiple jobs, e.g. for JDK 11 and 17, which otherwise is exactly the same.

  • By default, sh blocks imply -ex flags which means that any command exiting with non-zero code (error) will cause the whole block to be exited - and this will also fail the stage (unless caught). But if shebang is used (e.g. #!/bin/bash) the flags are not implied and must be added explicitly. It’s best to always write shebang as #!/bin/bash -ex or even with -eux if we want to enforce that all used variables are set (-x causes that each command is printed before execution). See the documentation for sh step. Alternatively, don’t use shebang unless really needed for some bash specific syntax.

  • jnlp container runs as jenkins user by default. For our purposes it’s better when it runs as root, so always add runAsUser: '0' to the definition. Typically, builder (Maven+JDK) and DB containers also run as root by default, but let’s use runAsUser everywhere. The reason for this is that some tools (e.g. npm) are sensitive when writing as root (from the builder image) to a directory that belongs to non-root owner.

  • Publisher step for TestNG result publication (results and chart) does NOT consider failed test configuration as a reason for unstable build. To change this, add failureOnFailedTestConfig: true to the configuration map for the step. See docs here.

See Also