In memory pubsub for bypassing redis

[ChaoticTempest]

Currently, it's kind of inefficient to insert into redis, and then immediately have the primitive (triples or presignatures) be taken immediately for processing. This creates latency where we need to send it over to the redis server, and then have it serialize to disk, and then deserialize back to MPC just to get the value.

Instead, I suggest we have a simple PubSub in memory where if somebody is already subscribed for that particular ID, we can just forward the primitive. We already have such a system with the message pubsub, but this would be simpler. Could just be map.

[volovyks]

Not sure if this happens often. Are you talking about newly generated tripels and pre-signatures and the fact that we can use them right away instead of saving and getting them from Redis?

In case of triples, I would say our long-term strategy should be "not to have them in Redis at all". Instead, save in RAM and use it for pre-signature generation.

Not saving primitives in storage can add a lot of complications for collecting statistics, Ts and Ps availability checks, etc.

[ChaoticTempest]

It happens pretty frequent. A proposing node wants to use a presignature but it hasn't completed generation on the other nodes yet. So this would be very helpful there. It's especially so I can eliminate the polling of redis for this presignature which also slows redis down because it needs to take up a connection and then check. There is a pubsub way we can do it via redis (blocking lists), but that still queries redis.

In case of triples, I would say our long-term strategy should be "not to have them in Redis at all". Instead, save in RAM and use it for pre-signature generation.

Not a bad idea for triples, but this requires a lot more work right now.

Not saving primitives in storage can add a lot of complications for collecting statistics, Ts and Ps availability checks, etc.

I don't see how it would. The metrics are logged for generation. The only metrics we have for redis is their latency for each operation but we never really bothered much with it. The availability doesn't get affected at all since these are inflight and would never have been seen in our metrics either ways.

[jakmeier]

Would this redis bypassing pub-sub mean we no longer persist all Ps and Ts we generated? Or would it be more of an optimization operating on the side?

I'm asking because anything only stored in memory is ephemeral. And the mpc specification as it is written now assumes we only act on persisted Ps and Ts.