Futures executor determinism within Simulator

[dnorman]

Deterministic execution is desirable and necessary for validation of the library and for experimentation.

This is to ensure that test cases reliably pass, and also to allow the creation of complex spatial data interchange scenarios for modeling purposes.

At present, the unbase::util::simulator::Simulator implementation does a reasonable job of ensuring that all messages are delivered within a specified instant, and that no resultant messages from those interactions are delivered until at least the next tick of the simulator clock ( if not later, as determined by the spatial coordinates of the origin/destination)

This functions using the following (pseudocode) loop:

• Determine next clock tick which contains events scheduled to be delievered
• Fetch a list of all events scheduled to be for that instant
• Deliver events from that list to their respective slabs in a deterministic order, ensuring delivery for all events is completed before moving on
• Update delivery stats (which are later used to determine if the simulator is quiescent)
• Repeat

Unfortunately for the Simulator, the full assimilation of such delivered events generally requires additional steps be taken by their respective slabs – The completion of which the simulator cannot, and should not wait for. This is because some of those steps require fetching additional data from remote Slabs. That requires sending and receiving additional messages, which requires the simulator to advance. If we waited for this before considering the delivery of the original event to be completed, the simulator would necessarily deadlock, because it cannot advance until delivery of ALL events for that instant are delivered to completion.

So, as a result, the Slab / impl SimEvent for MemoPayload has to signal completion of each delivery prior to awaiting anything which is related to the simulator – And commensurately, the follow-on steps have to be handled by a background task which is not managed by the Simulator.

This presents a problem, because now we have race conditions between the Futures executor for that task and the simulator.

As a temporary hack - The simulator presently contains several Delay::new(Duration::from_millis(100)).await; statements, which are strategically placed to allow the Futures executor to (probably) run to quiescence before the simulator attempts to deliver another message, or attempt to evaluate its own quiescence.

This is obviously terrible, and should be removed at the soonest opportunity.

So what shall we do?

Well, one option is to have the simulator do something like async_std::task::yield_now in place of those Delay statements. This ALLLmost works, because it it would reschedule the Simulator runner task to be after all other items in the task queue, which should achieve the same effect.

Unfortunately that won't quite do what we want, because:

• The executor is probably running in multi threaded mode. We could get around this by forcing everything to run in a single thread, but that sucks for performance reasons
• The simulator task won't stay at the back of the task queue, because as deliveries are made, additional tasks are put on the end of the queue, which we would want to ensure get executed first, before ticking the simulator clock and doing the next batch of deliveries.

It is conceivable that maybe you could run single threaded, and run async_std::task::yield_now both before AND after delivering each batch of events, which might have the desired effect, but again, at the cost of performance – Which could be a problem for larger simulations (> 10k slabs) which are planned.

I'm open to better ideas, but based on what I know so far:

I think the best approach is to implement a custom executor which allows a task to be scheduled for quiescent periods, so that it is ONLY executed when the task queue is totally empty. Under such a regime, the simulator would deliver a batch of events to completion – meaning that all relevant Slabs have superficially applied the events, and queued subsequent events to be evaluated by the Futures executor (this should already be the case).

Subsequently, those tasks will be processed to the greatest extent possible in the absence of new event deliveries ( including queuing new simulator events ) and only thereafter will the Simulator will tick again, delivering events for the next instant.