Documentation on How we Use the Collectors

[VincenzoFerme]

In the current, and possibly future setting, we assume that 1 collector collects data from one and only one service, even though they are implemented to be configurable and collect data from more services. This is: 1) control the performance impact, 2) isolate the information sent to the analysis part. We isolate the data using the container_name (the part before the _ that includes the information of the collector and the information about the name of the service from which we collect data from) and the host on which the container is deployed, because this couple of keys guarantee the unicity and the same container_name across the different trials of one experiment.

@Cerfoglg discuss about a possible extension to this, using a single collector for different containers. Probably we should also add the container_name of the monitored containers, removing from the same the trial_id that now it is part of the name to guarantee unicity in the container names we generate. This because then we need to identify all the data belonging to a certain experiment across all the trials of the experiment.

[Cerfoglg]

@VincenzoFerme

The collectors are currently implemented to assume the possibility of multiple containers to collect data from. If this happens to be the case, then the data is stored on Minio for each container, and when it's time to send the message to Kafka we still send a single message, but for the fields where we need data for each container we use a coma separated list to specify all of them. For example, the minio key field will now have multiple minio keys with comas as separator (Example: "minio/key/1,minio/key/2,minio/key/3"). Same applies to container ids and container names. The scheduler is already considering this possibility when receiving the messages. This may not be the ideal solution. Since we use JSON messages it would be better to marshall arrays directly instead of strings representing lists like this.

As for the container names, I don't know what's best overall. The fact remains that containers need to be identified across multiple trials, and since container ids change from one trial to another, the names are the only way to keep track of them.