Skip to content

Matrix job support

Jump to bottom
Sam Gleske edited this page Oct 4, 2017 · 48 revisions

What is a matrix job?

The concept of a matrix job is to test multiple types of similar technologies across different builds. Matrix builds are independent of each other so are run in parallel. For example, one might want to build their project testing across multiple versions of Java. Jervis builds against multiple versions of Groovy for backwards compatibility with older versions of Jenkins.

Environment matrix

Every project type has available to it the env YAML key. It's what Jervis uses to test multiple versions of Groovy. Travis CI also supports environment matrix. This allows launching similar builds with different sets of environment variables.

A simple one dimentional matrix can be tested with a list in the env key.

env:
  - ANIMAL=cat VEHICLE=bike_basket
  - ANIMAL=dog VEHICLE=car

Will execute two matrix builds in parallel. The first build will have environment variables ANIMAL=cat and VEHICLE=bike_basket set. The other build will have ANIMAL=dog and VEHICLE=car. Each build is called a matrix axis.

Sometimes, it is desireable to share environment variables across matrix axes. The following syntax is also supported for the env key. To run the same matrix builds above but share an environment variable across all matrix axes, the following syntax is support.

env:
  global:
    - DRIVER=person
  matrix:
    - ANIMAL=cat VEHICLE=bike_basket
    - ANIMAL=dog VEHICLE=car

Will execute two matrix builds in parallel. The first build will have environment variables ANIMAL=cat, VEHICLE=bike_basket, and DRIVER=person set. The other build will have ANIMAL=dog, VEHICLE=car, and DRIVER=person. In this case, DRIVER=person is shared across all buildable matrix axes.

Multidimensional matrix builds

To build multidimensional matrices you need to define matrix builds across multiple YAML keys. For example, Java supports matrix building for the env and jdk YAML key. Let's build a 2 dimensional matrix testing different environment configurations testing across multiple Java JDKs.

language: java
env:
  - DATABASE=mysql
  - DATABASE=postgres
  - DATABASE=sqlite
jdk:
  - openjdk7
  - openjdk8

The above will launch 6 matrix axis builds in parallel (a 2 dimensional 3x2 matrix) in order to test every possible combination.

Buildable matrix axes (6 builds in parallel):

  • env: DATABASE=mysql, jdk: openjdk7
  • env: DATABASE=postgres, jdk: openjdk7
  • env: DATABASE=sqlite, jdk: openjdk7
  • env: DATABASE=mysql, jdk: openjdk8
  • env: DATABASE=postgres, jdk: openjdk8
  • env: DATABASE=sqlite, jdk: openjdk8

A rule of thumb for calculating matrix building: for every YAML key which supports matrix building is an additional dimension possible for matrix building. Mathematically multiply together the number of values in each dimension to get the total possible builds run in parallel.

For example, the Ruby language supports env, rvm, gemfile, and jdk. In Ruby, it's possible to have a 4 dimensional matrix build.

Matrix filtering

With N-dimensional matrices in a build it is sometimes desirable to only build some matrix axis combinations and not others. Matrix filtering support allows developers to build only the matrix axes they need.

There are two ways to filter matrix axes. Filtering by include (whitelist) or filtering by exclude (blacklist). Filtering by include and exclude can be done at the same time. However, exclusion takes precedence over inclusion. That is, if an exclude filter matches a matrix axis then that axis will not be built regardless of include matching it.

Filter by exclude

Take for example this 3 dimensional matrix in ruby (4x3x4 matrix with maximum 48 buildable axes). In the following example, changing Java versions for MRI ruby doesn't make sense because it is not Java based. It should be excluded. After filtering there will only be 36 buildable axes.

language: ruby
rvm:
  - 1.9.2
  - jruby-18mode
  - jruby-19mode
  - jruby-head
jdk:
  - openjdk6
  - openjdk7
  - oraclejdk7
gemfile:
  - Gemfile
  - gemfiles/rails4.gemfile
  - gemfiles/rails31.gemfile
  - gemfiles/rails32.gemfile
matrix:
  exclude:
    - rvm: 1.9.2
      jdk: openjdk6
    - rvm: 1.9.2
      jdk: openjdk7
    - rvm: 1.9.2
      jdk: oraclejdk7

Shorthand exclusion is also possible. The following is an example of excluding all matrix indices which have rvm: 2.0.0 and gemfile: Gemfile no matter what other matrix dimension exists. The following filters down to 45 buildable matrix axes.

matrix:
  exclude:
    - rvm: 1.9.2
      gemfile: Gemfile

Filter by include

Filtering by include is a way to whitelist building matrix axes. Take the following example which is meant to be paired with the ruby example at the top of the Filter by exclude section.

matrix:
  exclude:
    - rvm: 1.9.2
      gemfile: Gemfile
  include:
    - rvm: 1.9.2

include adds to the exclude filter combining into a more comprehensive filter. The example above shows an exclude for rvm: 1.9.2 and gemfile: Gemfile. However, the include states rvm: 1.9.2. This means it will run only rvm: 1.9.2 (include) matrices except for when gemfile: Gemfile is matched (exclude). Put another way it can be thought of as the following groovy filter.

!(rvm == "1.9.2" && gemfile == "Gemfile") && (rvm == "1.9.2")

A thorough explanation and example is provided in this comment and this comment of issue #7.

Allow failures

This feature is not currently supported.

allow_failures has not been implemented in the matrix build plugin. There is an open feature request for matrixy project type JENKINS-26188 hoping to address this.

As pipeline support for Jervis expands, setting an unstable result for a buildable axis will be researched as an allowed failure rather than failing the whole pipeline.

Matrix support by language

This section includes quick links to languages and their matrix support. The languages only document tools typically customized for the build matrix of that language.

Note: other tool matrices not listed for the language can be specified as an additional matrix for a project. Large projects tend to consist of many languages, technologies, and stacks. Therefore, any tool can be grouped with any language. These are just common groupings. See Matrix support by tool for a comprehensive list of known supported tools.

Fully supported languages (code and wiki docs):

Nearly finished support (code but no wiki docs):

Unfinished support (no/some code and no wiki docs):

Android

For Android projects, env and jdk can be given as arrays to construct a build matrix.

C

For C projects, env and compiler can be given as arrays to construct a build matrix.

C++

For C++ projects, env and compiler can be given as arrays to construct a build matrix.

Clojure

For Clojure projects, env, lein, and jdk can be given as arrays to construct a build matrix.

TODO: research lein matrix support.

C Sharp (C#)

C# projects do not appear to have a matrix build currently.

D

For D projects, env and d can be given as arrays to construct a build matrix.

TODO: research d matrix support.

Erlang

For Erlang projects, env and otp_release can be given as arrays to construct a build matrix.

TODO: research otp_release matrix support.

F Sharp (F#)

F# projects do not appear to have a matrix build currently.

Go

For Go projects, env and go can be given as arrays to construct a build matrix.

TODO: research go matrix support.

Groovy

For Groovy projects, env and jdk can be given as arrays to construct a build matrix.

Haskell

For Haskell projects, env and ghc can be given as arrays to construct a build matrix.

TODO: research ghc matrix support.

Java

For Java projects, env and jdk can be given as arrays to construct a build matrix.

JavaScript (with Node.js)

For JavaScript/Node.js projects, env and node_js can be given as arrays to construct a build matrix.

TODO: research node_js matrix support.

Julia

For Julia projects, env and julia can be given as arrays to construct a build matrix.

TODO: research julia matrix support.

Objective-C

For Objective-C projects, env, rvm, gemfile, xcode_sdk, and xcode_scheme can be given as arrays to construct a build matrix.

TODO: research xcode_sdk and xcode_scheme matrix support.

Perl

For Perl projects, env and perl can be given as arrays to construct a build matrix.

TODO: research perl matrix support.

PHP

For PHP projects, env and php can be given as arrays to construct a build matrix.

TODO: research php matrix support.

Python

For Python projects, env and python can be given as arrays to construct a build matrix.

Ruby

For Ruby projects, env, rvm, gemfile, and jdk can be given as arrays to construct a build matrix.

Rust

Rust projects do not appear to have a matrix build currently.

Scala

For Scala projects, env, scala, and jdk can be given as arrays to construct a build matrix.

TODO: research scala matrix support.

Shell

Is a basic and empty language type defined to run scripts in the default shell of any selected platform and operating system. env can be given an array to construct a build matrix.

Visual Basic

Visual Basic projects do not appear to have a matrix build currently.

Matrix support by tool

The following documents matrix support by tool. In a language, multiple tools may be involved with building or testing that technology. The use of chaining tools for a particular language is called a toolchain.

compiler

Testing against two compilers will create (at least) 2 rows in your build matrix. e.g.

compiler:
  - clang
  - gcc

env

env support is documented at the top of this page under Environment matrix.

gemfile

Ruby can be built against multiple Gemfiles. It is useful to do this to test multiple versions of dependencies for your project (for example multiple rails versions).

language: ruby
gemfile:
  - Gemfile
  - gemfiles/rails4.gemfile
  - gemfiles/rails31.gemfile
  - gemfiles/rails32.gemfile

Will execute at least four matrix builds.

jdk

Testing with multiple JDKs. To test against OpenJDK 7 and OpenJDK 6:

jdk:
  - openjdk7
  - openjdk6

python

Virtualenv is what supports multiple versions of Python. By default builds will be run with Python 2.7. You can specify a version of Python using the python key in .jervis.yml.

python: "3.2"

To build against multiple versions of Python in a matrix build use a list.

python:
  - "2.6"
  - "2.7"

rvm

RVM is what supports multiple versions of Ruby in Jervis.

language: ruby
rvm:
  - 1.9.3
  - jruby-18mode # JRuby in 1.8 mode
  - jruby-19mode # JRuby in 1.9 mode
  - rbx-2.1.1
  - 1.8.7

Will execute at least five matrix builds. One for each version of ruby known to RVM. If a version of ruby is not installed then it will be installed automatically.

Table of Contents

User Guide

Admin Guide

Development Guide

Clone this wiki locally