message retry and WooFPut() cross-namespace semantics

[richwolski]

ZeroMQ recommends tearing down and re-establishing a socket connection that has timed out. The current WooFPut() semantics are that the caller gets back the sequence number of the put or an error.

With a proxy architecture for cross-namespace puts, these semantics make message retry in the event of a network partition difficult because the client can't know whether the put was lost or the reply was lost.

One way to fix this issue to allow the client to "know" whether the put "made it" is to implement a cross-namespace get that returns sequence numbers and a separate one that returns elements associated with specific sequence numbers. With these two calls a client could then use the following logic:

 get the highest sequence number
 seq_no = put(element)
 if(seq_no == ERROR) {
      get highest sequence number
      if highest sequence number didn't change {
            the put didn't make it
      } else {  // the reply was lost
            fetch the elements for the sequence numbers between first highest and this highest
            loop through elements to see if value is there
      }
 }

For now, we'll assume that ZeroMQ is being diligent about making socket connections but we'll need to fix this at some point.