v2 Tasks Design

Tasks Design

The document is partially outdated. Many things changed during the implementation.

This document was a draft for the implementation of the Tasks feature. During the implementation the design was adapted. The documents does not reflect the final implementation.

Problem Description

In a CI environment multiple tasks for an application are often run:

• Checks,
• Unit-Tests,
• Integration Tests,
• Build,
• etc

baur only supports to record build steps if they produce and upload an artifact. It's not possible to record a state for other tasks like tests or checks for a source version.

Ensuring that checks, tests, etc are only run if they haven't run before for an application version is only possible by running baur build as last step only if all previous steps were successful. This raises the following issues:

• build can not be run in parallel with other steps,
• all steps must be rerun if a build failed,
• baur can not be used to track tasks that do not build + upload an artifact, like running tests and checks for Go-Libraries.
• tests are not run in CI if only testfiles changed and they are not part of the build inputs

Solution

Allow to freely define tasks. A task definition consists of:

• a command to run (REQUIRED),
• input file specification (REQUIRED),
• output file specification (OPTIONAL)

baur records the results of the tasks for the source hash of the application in the database an provides commands to query the results. This would allow in a CI environment to e.g. query if a task called unit_tests already run successfully for the current source version.

Config Format

Reusability

To prevent that the same inputs, outputs and tasks definitions have to be maintained for multiple applications, those sections should be reusable.

Reusable sections can be specified in:

• the .app.toml file, this makes the sections only visible for the application or
• in *.toml files in include directories. The paths of the include directories can be specified in a setting called include_dirs in the .baur.toml file.

IDs for Input, Output and Task sections are unique per section. If multiple Input sections with the same name are defined an error is generated. Using the same name for an Input and an Output section is allowed though.

Inputs

Reusable Inputs can be specified in the following way:

[Includes]
  [[Input]]
    id = "input.gobuild_default" # identifies input section

    # [[Input]] can contain the same member then previously  [Build.Input]
    [Input.GolangSources]
      paths = ["."]
      environment = ["GOPATH=$ROOT/../../"]

    [Input.GitFiles]
      paths = ["store/postgres/migrations/", "$ROOT/bashbuild/main.inc",
               "$ROOT/bashbuild/inc/go", "$ROOT/bashbuild/inc/*.inc"]

  [[Input]]
    id = "input.gotest_default" # identifies input section

    # [[Input]] can contain the same member then previously  [Build.Input]
    [Input.GolangSources]
      paths = ["."]
      testfiles_only = true
      environment = ["GOPATH=$ROOT/../../"]

    [Input.GitFiles]
      paths = ["store/postgres/migrations/", "$ROOT/bashbuild/main.inc",
               "$ROOT/bashbuild/inc/go", "$ROOT/bashbuild/inc/*.inc"]

  [[Input]]
    id = "input.php_sources" # identifies input section

    [Input.GitFiles]
      paths = ["*.php"]

Outputs

Analogous reusable output sections are declared in the following format:

[Includes]
  [[Output]]
    id = "go_def_output"

    [Output.File]
      path = "dist/$APPNAME.tar.xz"

      [Output.File.S3Upload]
        bucket = "sisu-resources/binaries/sisu"
        dest_file = "$APPNAME-$GITCOMMIT.tar.xz"

Tasks

Reusable tasks are defined in the following format:

[Includes]
  [[Task]]
    id = "build.goapp"
    name = "build"
    command = "./build ci_build"
    includes = ["account_sources", "go_def_output"]

  # additional inputs and outputs can be added as [[Task.Input]] or
  # [[Task.Output]] sections
  [[Task.Input]]
    [[Task.Input.GitFiles]]
      paths = ["misc/"]

A task is included in an .app.toml file by refereeing it via it's id value. When a task is run via the baur CLI it is called by it's name value. name is also used to identify it in the task records. That allows to have e.g. different shared build apps by programming language but running them via the same command.

.app.toml

An exemplary .app.toml looks like:

name = "account-service"

[[Tasks]]
  tasks = ["build.goapp"]

  [[Tasks.Task]]
    name = "check"
    command = "./build check"
    includes = ["input.gobuild_default"]

  [[Tasks.Task]]
    name = "unit_test"
    command = "./build docker_test_unit"
    includes = ["input.gotest_default"]

Build Inputs

baur adds automatically all files that contain information for the task as it's input. This are the .app.toml and all included files.

Commandline Interface

baur run

Replaces baur build

baur run [--force] [--skip-upload] [([<APP>].<TASK>)|(<APP>[.<TASK>])]

If no TASK is specified, all tasks for the app are run. If no APP is specified, all apps that have a task with the provided name are run. By default only outstanding tasks are run.

baur ls tasks

Replaces baur ls apps

baur ls records

Replaces baur ls builds

baur ls records [--filter <FIELDNAME> <SPEC>] [--after <datetime>]
                [--before <datetime>] [--sort <FIELD>-<ORDER>] <APP>|all

baur show

baur show <APP<|<APP>.<TASK>|TASK-ID

Task-ID is the database record-id of a task.

baur ls apps

Is changed to show only static informations about applications:

baur ls apps [--abs-path] [--csv] --fields FIELD[,FIELD...] [<APP>]...

fields:
- app, path

baur status

baur status [flag]... [([<APP>].<TASK>)|(<APP>[.<TASK>])]...

flags:
--abs-path
--status
--task
--fields <FIELD>[,<FIELD>]... FIELDS is one of  app, path, id,
                                                status, record-git-commit

id is the database record id for the task, this might be confusing, same name then for include ids. (TODO)

Database Layout

pgmodeler file