Replace riak_core_mochiglobal with persistent_term #27
Description
The addition of persistent_term to OTP was intended to provide an alternative to the use of mochiglobal - https://www.erlang.org/blog/otp-22-highlights/#counteratomics-and-persistent_terms.
riak_core_mochiglobal is used only in riak_core_ring_manager - and the replacement for persistent_Term is straight-forward from a coding chnage perspective.
The caveat of persistent_term is the garbage_collection triggered by the update of a non-trivial term (which the Ring is):
When a (complex) term is deleted (using [erase/1](https://www.erlang.org/doc/apps/erts/persistent_term.html#erase/1)) or replaced by another (using [put/2](https://www.erlang.org/doc/apps/erts/persistent_term.html#put/2)), a global garbage collection is initiated. It works like this:
All processes in the system will be scheduled to run a scan of their heaps for the term that has been deleted. While such scan is relatively light-weight, if there are many processes, the system can become less responsive until all processes have scanned their heaps.
If the deleted term (or any part of it) is still used by a process, that process will do a major (fullsweep) garbage collection and copy the term into the process. However, at most two processes at a time will be scheduled to do that kind of garbage collection
The riak_core_ring_manager already has protection to mitigate this issue:
- the ring is only promoted to mochiglobal (i.e. in the future persistent_term) after a period of stability;
- In the "unstable" period all processes use an ETS cached version instead, and so when a new ring is promoted most processes no longer reference the old term (and so fullsweep will not generally be required).