ZeroMQ Island

[jdiez17]

At this point, the island is able to exchange solutions between peers, be embedded in archipelagos and act as a listener to monitor the status of a swarm of ZeroMQ islands. All of these capabilities are available in C++ (PaGMO) and Python (PyGMO).

Along with this code submission there is a set of static documents and IPython notebooks that describe the setup process, explain the island's API/user interface and intended use cases through examples, and show how to monitor the status of a ZeroMQ swarm. Additionally, there is language-specific documentation available for Python (in Sphinx) and C++ (in Doxygen).

The original project intended to use a brokerless approach for all network communications. We considered two approaches for brokerless operation: active, network-level broadcast messages (such as zeroconf), and passive port scanning. Due to the limitations of these techniques and the scope of cases in which islands can communicate, we chose to go with a centralised broker that is reachable by all nodes. An example of a situation in which active discovery or passive scanning is unsuitable: inter-datacenter communication.

The original intent was to write a custom broker in ZeroMQ/C++, but due to time constraints and other factors the current ZeroMQ island uses Redis as a broker. Replacing it would not be complicated, because the broker is only maintaining a map of channel tokens to a list of IP addresses and providing realtime communication. Both of these things are relatively simple uses of the different types of sockets that ZeroMQ provides.

Because we are using an external broker, some dependencies are added to the project. These dependencies are completely optional and will not affect users that choose to compile PaGMO without ZeroMQ support (this is the default behaviour).

A simple example:

from PyGMO import *
import time

prob = problem.schwefel(10)
algo = algorithm.de(10)
pop = population(prob, 20)

isl = zmq_island(algo, pop)
isl.set_broker_details("127.0.0.1", 6379)
isl.set_token("schwefel10_de10_pop20")
isl.set_ip("127.0.0.1")
isl.connect()

while True:
    isl.evolve(10)
    print("Best: ", isl.population.champion.x)

    time.sleep(1)

[gnu-user]

Is there any chance that this code can be merged into master, the addition of ZeroMQ would make this very applicable for clustered environments.

[darioizzo]

We are currently focussing our energies on an overhaul of PaGMO that should also solve its applicability to clustered environments. But it will take a while ....