Implement periodic saving for Storage and EventManager

[jcarraror]

• Motivation: Currently, all blocks of the blockchain are being stored to the memory, we must add functionality to periodically dump the blocks into the database in order to not overload the system memory with blocks

• Objectives: Periodically dump the most old unused blocks from memory to the database

• Success Criteria: Safely dump old blocks to the database, not locking Storage for too long (more than one second) while dumping.

• Dependencies: No external dependencies.

• Risks and Mitigations: It is necessary to be aware of blocks being a shared_ptr’s, meaning that while it is being used by another thread, it should not be dealocated from memory, besides that, it should not lock the storage for too long where it would create a deadlock within the node, limiting it’s response for processing new blocks or requests from other nodes, besides that, priority should be given to the older blocks within the deque.

• Resource Allocation: This milestone will deeply affect the Storage function

[Jean-Lessa]

Branch 27-periodic-save (commit 99a6f35) contains a prototype for a possible implementation of Storage::periodicSaveToDB(). Heavy emphasis on prototype.

The thread is already set to start at the end of Storage's constructor and stop at the start of its destructor. It starts just fine and the cooldown is iterated upon as well (every 15 seconds), but for some reason I don't see the message about stopping the thread when I kill tmux. Unless I'm missing something here, that means the destructor is not being called at all, which also means the blockchain is never really being saved at all, independent of the periodic thread, and this actually concerns me since it practically means we have no idea if the destructor is actually working...? Either way, you can check this yourselves in debug.txt, filtering by "periodic".

Basic logic for the saving routine is as follows (subject to change, could use some input on the whole thing, including testing which was not done properly due to lack of feedback on how to properly test it):

• Next cooldown loop: check if cachedBlocks has at least 1000 blocks, or cachedTxs has at least 1M (1 million) transactions - start the saving routine if that's the case, skip it and wait 15 seconds for the next loop otherwise
• Set a time point at the start of the routine and calculate 50% of the chain's size to save to the DB (e.g. 500 blocks if chain has 1000 blocks)
  • In the rare event that the chain itself has less than 2 blocks, we skip the routine and loop the next cooldown (it could happen? I mean the cooldown loop checks the cache which comes from DB, not the chain which comes from memory, also if a single block has a chance of having 1M txs it could trigger the routine)
  • This means the saving routine will only really happen if there's at least 2 blocks in chain (this is probably an unnecessary edge case check, needs further analysis)
• Do the same saving routine as the destructor (remove block tx references and the blocks themselves from maps and the chain, put them on batches, save them all at the end), always saving older blocks first, but adapted to a few constraints:
  • Check first if the block is unique to the Storage class via shared_ptr::use_count() (check the code comments for more info) - if not, wait until it is (checking every 1ms on a loop)
  • If the routine takes more than 1 second to complete, it is aborted prematurely and whatever blocks were retrieved until that point within the batches are saved (this is due to the mutex locking the whole chain, as explained at the start of the issue)
  • e.g. saving 500 blocks, 1 second later we only batched 100 because block 101 is not unique and it stalled there until the end - abort, save those 100 blocks, then wait for next cooldown loop
  • This presents a (very rare but still possible) annoyance - if block 1 somehow always stalls until the end on every cooldown loop, saving is delayed ad infinitum - this could be prevented with a better but more complex logic (or tweaking the waiting time as I'm not sure 1ms is too little or too much), but I find that really hard to happen as far as I'm concerned

How could this be properly tested? The routine only kicks in at 1000 blocks or 1M txs on cache (which is also weird because we should be checking the chain and I still don't get that, but I strictly followed the TODO instructions anyway), so we'd need to check the following steps:

• Deploy the blockchain and monitor debug.txt all the way
• Fill the blockchain with thousands of blocks (or a couple blocks with millions of txs) in the span of 15 seconds or so
• Let the thread do its magic
• Repeat steps 2 and 3 a few times maybe
• Shutdown the blockchain, start it up again and see if blocks are properly loaded from the DB into memory