Feedback - Docker-Compose Playground

[FreeSoftwareServers]

Greetings!

I'm mostly just here to give some feedback as a completely new user to icinga2 and my experience during setup/testing so far. At this point I have the stack running in compose the way I want and have configured e-mails, my next steps are too add a remote host and test using a custom bash script to return a check/notification, but I expect that part to be easier and less docker related so this post is mainly focused on getting the stack running in docker-compose.

I'm coming from xymon history but I'm unimpressed that xymon is hosted on sourcefourge, doesn't have a docker image and looks dated, but I do love my xymon! I saw a little comment on sourceforge/interwebs that somebody whom previously used xymon rec icinga and I gave it a go. It certainly has a beautiful UI and code on Git and a proper docker-image which checked those boxes I wanted!

As far as feedback goes:

• I didn't like to setup of all the env vars in the compose file, it was too hard to understand what belong to where and so many repeating identical variables made it hard to understand how to change something. But, it was nice that I was able to clone/run git repo without any issues. Maybe a second playground would be what I'm suggesting as I think its preferred to have new users use the Wizard/Web-UI during setup to understand configs/components and validate options etc.
• I have very few env vars in my compose, mostly just DB name/pass.
• I have a separate DB container for each DB, I feel this is easier to read/understand/setup in compose and has the ability to restart/edit/mess with one DB without touching them all
• I find many projects are moving towards just pgsql support, I might recommend going with the flow on this and at least starting to use it as default in configs, while still supporting mysql till a major version change like v3 or v4 etc.
• I've got director setup and I'm starting to use it, but one thing I noticed is that there are no notes anywhere about getting e-mail working, I was able to get it working by simply binding msmtprc and aliases but this should be documented. I then noticed that the conf.d/ dir was full of configs that the director didn't seem to know about and the system was trying to e-mail icinga@localhost which I adjusted via CLI to be my e-mail, but this seems like should be perhaps I should clear out this directory and setup 100% via director? I'm new and haven't wiped dir yet, just something I'm thinking to try
• I realize that you don't rec docker for prod, but many (me included) may decide that the benefits outweigh the negatives, at least initially. As well, its much easier for me to add to existing hosts vs ask mgmt for a new dedicated hardware box (big corp....). Perhaps another repo not called "playground"?
• I struggled to get incinga2db/redis setup via UI and had to config via CLI, not sure why.
• I like to move all env vars into .env
• A simple note on testing API would be nice in docs, I use GET url:port/v1/objects/hosts
• It would be nice to have a "story" setup, something like, get stack running, send forced e-mail notification test, remove/ack useless container checks that fail as they are checking inside container, add docker HOST as agent, add custom check w/ bash and have it fail/pass and notify.

Anyway, thought I'd share my configs where I ended at, let me know if I can help more/clarify any of my points etc.

Steps:

• Run Stack and Run Web-Wizard
• Get Setup Token
  docker exec -it icinga2-web /bin/bash -c '/usr/share/icingaweb2/bin/icingacli setup token create'
• Get Initial Password (change later)
  docker exec -it icinga2-api /bin/bash -c 'cat /etc/icinga2/conf.d/api-users.conf'
• Note Skip Validation w/ Redis that Icinga is connected as well as we fix after wizard
• Edit API User and InciaDB config files and root user via UI
• Restart Stack
• Go to URL/config/resource and add DirectorDB As Resource
• Restart Stack

docker-compose.yaml:

WD=/opt/icinga
#rm $WD -R
mkdir -p $WD/{setup,conf,conf/web,conf/api,conf/msmtp,sql,sql/icinga,sql/web,sql/director,redis}
cd $WD/setup

cat << 'EOF' >docker-compose.yaml 
version: '3.7'
services:

  icinga2-web:
    image: icinga/icingaweb2
    hostname: icinga2-web
    container_name: icinga2-web
    depends_on:
      - icinga2-api
      - icinga2-webdb
      - icinga2-db
      - icinga2-icingadb
      - icinga2-redis
    ports:
      - '8888:8080'
    networks:
      - icinga-net
    volumes:
      - type: volume
        source: icinga-web
        target: /data

  icinga2-web_director:
    image: icinga/icingaweb2
    command: icingacli director daemon run
    hostname: icinga2-web_director
    container_name: icinga2-web_director
    depends_on:
      - icinga2-web
    networks:
      - icinga-net
    volumes:
      - type: volume
        source: icinga-web
        target: /data

  icinga2-webdb:
    image: mariadb:10.7
#    image: postgres
    hostname: icinga2-webdb
    container_name: icinga2-webdb
    networks:
      - icinga-net
    environment:
#      - 'POSTGRES_DB=${WEB_DB_NAME}'
#      - 'POSTGRES_USER=${WEB_DB_USER}'
#      - 'POSTGRES_PASSWORD=${WEB_DB_PWD}'
#      - 'POSTGRES_ENCODING=UTF8'
 #     - 'MYSQL_ROOT_PASSWORD=${WEB_DB_PWD}'
      - 'MYSQL_RANDOM_ROOT_PASSWORD=1'
      - 'MYSQL_DATABASE=${WEB_DB_NAME}'
      - 'MYSQL_USER=${WEB_DB_USER}'
      - 'MYSQL_PASSWORD=${WEB_DB_PWD}'
    volumes:
      - type: volume
        source: icinga-webdb
        target: /var/lib/mysql

  icinga2-api:
    image: icinga/icinga2
    hostname: icinga2-api
    container_name: icinga2-api
    ports:
      - '5665:5665'
    networks:
      - icinga-net
    environment:
      - 'ICINGA_MASTER=${ICINGA_MASTER}'
    volumes:
      - type: volume
        source: icinga-api
        target: /data
      - type: bind
        source: /opt/icinga/conf/msmtp/msmtprc
        target: /etc/msmtprc
      - type: bind
        source: /opt/icinga/conf/msmtp/aliases
        target: /etc/aliases

  icinga2-icingadb:
    image: icinga/icingadb
    hostname: icinga2-icingadb
    container_name: icinga2-icingadb
    networks:
      - icinga-net
    environment:
      - 'ICINGADB_DATABASE_HOST=icinga2-db'
      - 'ICINGADB_DATABASE_PORT=${ICINGA_DB_PORT}'
      - 'ICINGADB_DATABASE_DATABASE=${ICINGA_DB_NAME}'
      - 'ICINGADB_DATABASE_USER=${ICINGA_DB_USER}'
      - 'ICINGADB_DATABASE_PASSWORD=${ICINGA_DB_PWD}'
      - 'ICINGADB_REDIS_HOST=icinga2-redis'
      - 'ICINGADB_REDIS_PORT=6380'

  icinga2-redis:
    image: redis:7.0.4
    hostname: icinga2-redis
    container_name: icinga2-redis
    command: --port 6380
    networks:
      - icinga-net
    volumes:
      - type: volume
        source: icinga-redis
        target: /data

  icinga2-db:
    image: mariadb:10.7
#    image: postgres
    hostname: icinga2-db
    container_name: icinga2-db
    networks:
      - icinga-net
    environment:
#      - 'POSTGRES_DB=${ICINGA_DB_NAME}'
#      - 'POSTGRES_USER=${ICINGA_DB_USER}'
#      - 'POSTGRES_PASSWORD=${ICINGA_DB_PWD}'
#      - 'POSTGRES_ENCODING=UTF8'
 #     - 'MYSQL_ROOT_PASSWORD=${ICINGA_DB_PWD}'
      - 'MYSQL_RANDOM_ROOT_PASSWORD=1'
      - 'MYSQL_DATABASE=${ICINGA_DB_NAME}'
      - 'MYSQL_USER=${ICINGA_DB_USER}'
      - 'MYSQL_PASSWORD=${ICINGA_DB_PWD}'
    volumes:
      - type: volume
        source: icinga-db
        target: /var/lib/mysql

  icinga2-directordb:
    image: mariadb:10.7
    command: --character-set-server=utf8 --collation-server=utf8_general_ci
#    image: postgres
    hostname: icinga2-directordb
    container_name: icinga2-directordb
    networks:
      - icinga-net
    environment:
#      - 'POSTGRES_DB=${DIRECTOR_DB_NAME}'
#      - 'POSTGRES_USER=${DIRECTOR_DB_USER}'
#      - 'POSTGRES_PASSWORD=${DIRECTOR_DB_PWD}'
#      - 'POSTGRES_ENCODING=UTF8'
 #     - 'MYSQL_ROOT_PASSWORD=${DIRECTOR_DB_PWD}'
      - 'MYSQL_RANDOM_ROOT_PASSWORD=1'
      - 'MYSQL_DATABASE=${DIRECTOR_DB_NAME}'
      - 'MYSQL_USER=${DIRECTOR_DB_USER}'
      - 'MYSQL_PASSWORD=${DIRECTOR_DB_PWD}'
    volumes:
      - type: volume
        source: icinga-directordb
        target: /var/lib/mysql

volumes:
  icinga-web:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/conf/web'
  icinga-webdb:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/sql/web'
  icinga-api:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/conf/api'
  icinga-redis:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/redis'
  icinga-db:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/sql/icinga'
  icinga-directordb:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/opt/icinga/sql/director'

networks:
  icinga-net:
    driver: bridge
EOF

env:

cd $WD/setup
cat << 'EOF'>.env
TZ=America/Whitehorse

#ICINGA_CONF
ICINGA_MASTER=1

#ICINGA_WEB
WEB_DB_NAME=icinga2web
WEB_DB_USER=icinga2
WEB_DB_PWD=icinga2
WEB_DB_PORT=3306

#ICINGA_DB
ICINGA_DB_NAME=icinga2db
ICINGA_DB_USER=icinga2
ICINGA_DB_PWD=icinga2
ICINGA_DB_PORT=3306

#ICINGA_DIRECTOR
DIRECTOR_DB_NAME=icinga2director
DIRECTOR_DB_USER=icinga2
DIRECTOR_DB_PWD=icinga2
DIRECTOR_DB_PORT=3306

EOF

cat <<'EOF'>/opt/icinga/conf/api/etc/icinga2/conf.d/api-users.conf
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "icinga2"
  client_cn = "icinga2-api"

  permissions = [ "*" ]
}
EOF

cat <<'EOF'>/opt/icinga/conf/web/etc/icingaweb2/modules/icingadb/commandtransports.ini
[icinga2]
skip_validation = "0"
transport = "api"
host = "icinga2-api"
port = "5665"
username = "root"
password = "icinga2"
EOF

cat <<'EOF'>/opt/icinga/conf/api/etc/icinga2/features-available/icingadb.conf
object IcingaDB "icingadb" {
  host = "icinga2-redis"
  port = 6380
  //password = "xxx"
}
EOF

cd /opt/icinga/conf/api/etc/icinga2/features-enabled
ln -s ../features-available/icingadb.conf .
docker restart icinga2-api
docker restart icinga2-web

email:

cat <<'EOF'>msmtprc
# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log
aliases        /etc/aliases

# Gmail
account        GMail
host           smtp.gmail.com
port           587
from           freesoftwareservers@gmail.com
user           freesoftwareservers
password       PASSWD

# PLEASE SET THIS LINE
account default : GMail
EOF
chmod 755 msmtprc

cat <<'EOF'>aliases 
default : freesoftwareservers@gmail.com
icinga : freesoftwareservers@gmail.com
root : freesoftwareservers@gmail.com
EOF
chmod 755 aliases

testing:

mailx -r freesoftwareservers@gmail.com -s "MailX Test" freesoftwareservers@gmail.com < /dev/null && sleep 2 && cat /var/log/msmtp.log

'/etc/icinga2/scripts/mail-service-notification.sh' '-4' '127.0.0.1' '-6' '::1' '-b' '' '-c' '' '-d' '2022-09-11 17:33:18 +0000' '-e' 'ssh' '-l' 'icinga2-api' '-n' 'icinga2-api' '-o' 'connect to address 127.0.0.1 and port 22: Connection refused' '-r' 'icinga@localhost' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'ssh' '-v' 'false'

'/etc/icinga2/scripts/mail-host-notification.sh' '-4' '127.0.0.1' '-6' '::1' '-

Here is where I keep my updated notes:

https://www.freesoftwareservers.com/display/FREES/ICINGA
https://www.freesoftwareservers.com/display/FREES/ICINGA-+Docker-Compose+-+Prod
https://www.freesoftwareservers.com/display/FREES/ICINGA+-+Docker+-+E-Mail+Setup

[a1ad]

Thanks for this.

Everything looks good, except I get the error msg: "Can't connect to Icinga Redis: Cannot assign requested address [tcp://localhost:6380]" in the web interface.

Why is he trying to connect to localhost when the redis host "icinga2-redis" is set in: /opt/icinga/conf/api/etc/icinga2/features-available/icingadb.conf ?

cat conf/api/etc/icinga2/features-available/icingadb.conf
object IcingaDB "icingadb" {
  host = "icinga2-redis"
  port = 6380
  //password = "xxx"
}

icinga2-redis | 1:M 05 Dec 2022 09:01:06.313 * Ready to accept connections

icinga2-icingadb      | 2022-12-05T09:01:07.932Z	INFO	icingadb	Starting Icinga DB
icinga2-icingadb      | 2022-12-05T09:01:07.932Z	INFO	icingadb	Connecting to database at 'icinga2-db:3306'
icinga2-icingadb      | 2022-12-05T09:01:07.935Z	INFO	icingadb	Connecting to Redis at 'icinga2-redis:6380'
icinga2-icingadb      | 2022-12-05T09:01:10.936Z	INFO	icingadb	Waiting for Icinga 2 to write into Redis, please make sure you have started Icinga 2 and the Icinga DB feature is enabled

Edit: I am going to write down the things i stumble on for future readers:
I needed to change the config here:

Next problem:

So i did the api thing

docker exec -it icinga2-api /bin/bash -c 'icinga2 node setup --master --zone master --cn icinga2-api --listen icinga2-api,5665 --disable-confd'
information/cli: Checking in existing certificates for common name 'icinga2-api'...
warning/cli: Certificate '/var/lib/icinga2/certs//icinga2-api.crt' for CN 'icinga2-api' already exists. Not generating new certificate.
information/cli: Generating master configuration for Icinga 2.
information/cli: API user config file '/etc/icinga2/conf.d/api-users.conf' already exists, not creating config file.
information/cli: Reading '/etc/icinga2/icinga2.conf'.
information/cli: Updating '"conf.d/api-users.conf"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Created backup file '/etc/icinga2/icinga2.conf.orig'.
information/cli: Include statement 'include "conf.d/api-users.conf"' already set.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Generating zone and object configuration.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
information/cli: Updating the APIListener feature.
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating 'NodeName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating 'ZoneName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating 'TicketSalt' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Edit the api feature config file '/etc/icinga2/features-available/api.conf' and set a secure 'ticket_salt' attribute.
information/cli: Updating '"conf.d"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Backup file '/etc/icinga2/icinga2.conf.orig' already exists. Skipping backup.
warning/cli: Tried to disable conf.d inclusion but failed, possibly it's already disabled.
information/cli: Updating '"conf.d/api-users.conf"' include in '/etc/icinga2/icinga2.conf'.
information/cli: Backup file '/etc/icinga2/icinga2.conf.orig' already exists. Skipping backup.
information/cli: Include statement 'include "conf.d/api-users.conf"' already set.
information/cli: Make sure to restart Icinga 2.

But there is no active API port
API config:

 cat conf.d/api-users.conf 
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "icinga2022"
  client_cn = "icinga2-api"

  permissions = [ "*" ]
}

Tests from the web container to API:

root@icinga2-web:/# nmap icinga2-api -p 5665
Starting Nmap 7.80 ( https://nmap.org ) at 2022-12-05 10:26 UTC
Nmap scan report for icinga2-api (192.168.96.3)
Host is up (0.000070s latency).
rDNS record for 192.168.96.3: icinga2-api.icinga2_icinga-net

PORT     STATE  SERVICE
5665/tcp closed unknown
MAC Address: 02:42:C0:A8:60:03 (Unknown)

root@icinga2-web:/# curl -k -u root:icinga2022 https://icinga2-api:5665/v1/objects/hosts
curl: (7) Failed to connect to icinga2-api port 5665: Connection refused

So, i guess the API thing is not working in my setup.. @FreeSoftwareServers any idea?

[back-2-95]

@a1ad did you get the setup working? I see all the "freesoftwareservers" links to any notes don't work anymore.