Icingacli checks

[t3easy]

BTW.: do you have a good idea for icingacli checks? As web and icinga are splitted, the checks can't run as there is no icingacli.

Maybe a Phar of the icingacli plus configuration deployed to the Icinga container. But I don't know if creating the Phar works that easily.

Originally posted by @lippserd in #17 (comment)

I just had an idea how to run such tests.
As there is a by_ssh command, that runs commands on a remote server by ssh, we could implement a by_docker command that can run check commands inside an existing container by docker exec icingaweb icingacli ... or by creating a new container with a custom check command docker run --rm check_xy_image.

Therefore the docker cli in the icinga2 image and the docker socket of the host of the icinga2 service or a ssh/tls DOCKERHOST is required.

Its not perfect for all hosting environments but the best solution for common.

[t3easy]

Here what I did so far as a first quick but working solution:

ARG ICINGA2_VERSION==2.13.5
ARG DOCKER_VERSION=20.10

FROM docker:${DOCKER_VERSION} as docker-cli

# Icinga 2
FROM icinga/icinga2:${ICINGA2_VERSION} as icinga2
USER root
COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
RUN set -eux; \
        groupadd -g 999 docker; \
        usermod -aG docker icinga;
USER icinga

services:
  icinga2:
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock

object CheckCommand "by_docker" {
    import "plugin-check-command"
    command = [ "/usr/local/bin/docker" ]
    arguments += {
        docker_command = {
            order = 10
            required = true
            skip_key = true
            value = {{
                if (macro("$by_docker_run$")) {
                return "run"
            } else {
                return "exec"
            }
            }}
        }
        docker_container = {
            description = "The container in that the command will be executed"
            order = 20
            required = true
            skip_key = true
            value = "$by_docker_container$"
        }
        docker_container_command = {
            description = "The command that will be executed in the container"
            order = 30
            required = true
            skip_key = true
            value = "$by_docker_container_command$"
        }
        docker_container_command_arguments = {
            description = "The arguments for the command that will be executed in the container"
            order = 40
            skip_key = true
            value = "$by_docker_container_command_arguments$"
        }
    }
}

template Service "Icingacli by Docker" {
    check_command = "by_docker"
    check_interval = 1m
    retry_interval = 30s
    vars.by_docker_container = "monitoring-icingaweb2-1"
    vars.by_docker_container_command = "/usr/local/bin/icingacli"
}

[nootnooooot]

In my environment we weren't so keen on exposing the docker socket to the containers, so I went down the rabbit hole of trying to patch the icingacli into the icinga2 container.
I did get this working, but ultimately it started to feel like an ugly solution as well and I went another route. However I thought I'd post what I have for anyone else that wants to go down this path.

Note that I'm by no means a Docker expert, and my strategy was to add almost everything then trim down the unnecessary parts. I never got to the point where I trimmed it down, so I can't say how much of this is actually required. Use at your own risk etc.

# Get icingaweb image to extract icingacli
FROM icinga/icingaweb2 as extract

# Mount base image
FROM icinga/icinga2

# Switch to root
USER root

...other icinga2 Docker code...

# ------------------------------------ Install IcingaCLI ------------------------------------ #
#Add Repositories
RUN echo "deb http://deb.debian.org/debian bookworm main contrib non-free" > /etc/apt/sources.list \
    && echo "deb http://security.debian.org/debian-security bookworm-security main contrib non-free" >> /etc/apt/sources.list \
    && echo "deb http://deb.debian.org/debian bookworm-updates main contrib non-free" >> /etc/apt/sources.list

# Install necessary packages
RUN apt-mark hold ca-certificates
RUN export DEBIAN_FRONTEND=noninteractive && \
    apt-get update && \
    apt-get upgrade -y || true && \
    apt-get install --no-install-recommends --no-install-suggests -y \
    openssl \
    libapache2-mod-php8.2 \
    libldap-common \
    locales-all \
    php-imagick \
    php-redis \
    php8.2-bcmath \
    php8.2-bz2 \
    php8.2-common \
    php8.2-curl \
    php8.2-dba \
    php8.2-enchant \
    php8.2-gd \
    php8.2-gmp \
    php8.2-imap \
    php8.2-interbase \
    php8.2-intl \
    php8.2-ldap \
    php8.2-mbstring \
    php8.2-mysql \
    php8.2-odbc \
    php8.2-opcache \
    php8.2-pgsql \
    php8.2-pspell \
    php8.2-readline \
    php8.2-snmp \
    php8.2-soap \
    php8.2-sqlite3 \
    php8.2-sybase \
    php8.2-tidy \
    php8.2-xml \
    php8.2-xmlrpc \
    php8.2-xsl \
    php8.2-zip && \
    apt-get clean && \
    rm -vrf /var/lib/apt/lists/*

#Copy required files for icingacli
COPY --from=extract /usr/share/icingaweb2 /usr/share/icingaweb2
RUN ln -s /usr/share/icingaweb2/bin/icingacli /usr/local/bin/icingacli
COPY --from=extract /usr/share/icinga-php /usr/share/icinga-php
COPY --from=extract /etc/php/8.2/cli/conf.d/99-docker.ini /etc/php/8.2/cli/conf.d/99-docker.ini

# Set permissions
RUN chown -R icinga:icinga /usr/share/icingaweb2
RUN chown -R icinga:icinga /etc/php/8.2/cli/conf.d/99-docker.ini
RUN chown -R icinga:icinga /usr/share/icinga-php
# ------------------------------------------------------------------------------------------- #

USER icinga