Add logging to native health check

[fsamuel-bs]

We fail after 2 minutes when checking the native health check.
This might not be enough time to have all the docker containers up.
We can configure the timeout, but it was quite hard to debug it without logging.

[fsamuel-bs]

Having this here is sad. Should I make ClusterHealthCheck an abstract class and have this as a private variable?

[j-baker]

Nope - instead, note that this is only used in NativeHealthCheck and do

static ClusterHealthCheck nativeHealthChecks() {
    return new ClusterHealthCheck() {
        static final Logger log = LoggerFactory.getLogger(ClusterHealthCheck.class);
        @Override
        SuccessOrFailure isClusterHealthy(Cluster cluster) throws InterruptedException {
            Set<String> unhealthyContainers = new LinkedHashSet<>();
            .........
        }
    }
}

[j-baker]

these may be a little spammy, FYI. Think this is done in a fairly tight loop, so you may end up with many log lines.

Instead, I'd recommend doing this like:

Set<String> lastUnhealthyContainers = new LinkedHashSet<>();
return new ClusterHealthCheck() {
    @Override
    public SuccessOrFailure isClusterHealthy(Cluster cluster) throws InterruptedException {
        Set<String> currentUnhealthyContainers = new LinkedHashSet<>();
        boolean healthyContainerSetChanged = false;
        for (Container container : cluster.allContainers()) {
            State state = container.state();
            if (state == State.UNHEALTHY) {
                currentUnhealthyContainers.add(container.getContainerName());
            }
        }

        if (currentUnhealthyContainers.equals(lastUnhealthyContainers)) {
              log(currentUnhealthyContainers);
              lastUnhealthyContainers.clear();
              lastUnhealthyContainers.addAll(currentUnhealthyContainers);
        }

        if (!currentUnhealthyContainers.isEmpty()) {
            return SuccessOrFailure.failure("The following containers are not healthy: " ...);
        }

        return SuccessOrFailure.success();
    }
}

[j-baker]

where you only log if the state has actually changed.