Parallel processing

Plan is to use Twisted, a comprehensive Python library for networking. It also has side projects for embedding a web server and doing server method calls from clients.

By default MDiG will run as a server and spawn a client to run locally. Further clients can be run by specifying as such on the command line and providing the server IP address and port.

Only the server provides a web interface, but it displays information on all clients.

There needs to be a mechanism for transferring GRASS files between the server and client. The Server will run in the location/mapset of the simulation and the the clients will construct new locations and mapsets and copy necessary files from the server... and then copy them back once processing is complete.

On client connection:

1. Check client has necessary executables (those that are listed in the xml file... not necessarily ALL MDiG modules).
2. Client creates new temp location based on information from the server.
3. For each task, create new mapset, copy required maps. Run sim/task. On completion return results to server.
4. Ping server about next task, either:
   1. analyse recently run replicate (to avoid resending simulation results).
   2. rerun simulation, create new replicate
   3. run different experiment

Other tasks to parallise are:

• generating envelopes. Split each year being combined into an envelope into a separate task.
• running analysis on envelopes.
• running analysis on a replicate (that was simulated on different client).

Sidenote: twisted.plugins could be used to define an interface for further python based modules.