Stream2 START and END messages should be pushed to all receivers

[sdebionne]

It would make the receivers logic much easier (no inter-receiver communication required) if all PULL sockets get a copy of the START and END messages. For this, the DCU would need to push n x START and END messages (where n is the number of receivers connected).

Currently the behavior is the following for an acquisition of 3 images with 3 receivers:

rcv 0 START (get frame info)
rcv 1 img1  
rcv 2 img2
rcv 0 img3
rcv 1 END

[GDYendell]

I don't think this would help because a one PULL client could get two of the start/end messages.

[sdebionne]

It's not clear to me when this would happen since the ZMQ doc state:

The PUSH socket type talks to a set of anonymous PULL peers, sending messages using a round-robin algorithm.

I did not encounter this situation so far (the msg are correctly round-robin dispatched). Did you?

If it can happen then I agree that this would not help.

[GDYendell]

Hmm. I may be wrong, but I don't think it is guaranteed and depends on load of the workers. Although it might just be the initial messages before the server realises that there are two clients and then it balances out.

[sdebionne]

It is what I suspect when reading

The PULL socket type talks to a set of anonymous PUSH peers, receiving messages using a fair-queuing algorithm.

where

Fair queuing uses one queue per packet flow and services them in rotation, such that each flow can "obtain an equal fraction of the resources".[1][2]

Depending on the implementation (byte-weighted fair?), the fair-queuing might interfere with the dispatching of the messages. Taking this into account in our design is not trivial.

Do you use multiple clients (multiple processes) at DLS?

[GDYendell]

We do, but ever since the first eiger we have had a single application that sits in between the detector and the multiple writers. This is because we wanted to write in sequential blocks of 1000 frames (for example, it is configurable), so we have never had to deal with this problem (if it is a problem).

I would very be surprised if fair-queuing in the context of zmq was byte-weighted rather than message-weighted. So, it might be OK, but you will definitely have to handle the issue of late joiners. This is easy to reproduce as a problem if you have a push server with two PULL clients and recv immediately after connection.

I have written some simple python scripts to test and it appears that as long as there is a pause between connect and sending data, two clients get the same number of messages. So I think I am wrong on this - sorry about that. You might need to stress test this to convince yourself.

# push.py
import zmq

context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind("tcp://127.0.0.1:5557")

while True:
    input()
    for num in range(1000):
        print(f"Sending {num}")
        data = {"num": num}
        socket.send_json(data)

# pull.py
import zmq

context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.connect("tcp://127.0.0.1:5557")

count = 0
while True:
    data = socket.recv_json()
    print(f"Received {data['num']} - {count}")
    count += 1

Output:

...
Received 988 - 95494
Received 990 - 95495
Received 992 - 95496
Received 994 - 95497
Received 996 - 95498
Received 998 - 95499

and

Received 989 - 95494
Received 991 - 95495
Received 993 - 95496
Received 995 - 95497
Received 997 - 95498
Received 999 - 95499

[sdebionne]

No need to be sorry, and thank you so much for investigating! For now, I'll add the necessary interprocess communication to synchronize on start and end messages. I'll report if we find that some images are not round-robin dispatched.