Make dynamic service dependency persistence #38
Description
Backgroud
Metric source for lazyload: Prometheus and AccessLog.
In Prometheus mode, the data is stored in Prometheus and the LazyLoad Controller queries the results in real time without storing this data, so this mode does not require a persistence transformation.
In AccessLog mode, the global-sidecar sends the AccessLog containing the service dependencies to the LazyLoad Controller after it has completed the underhanded forwarding. The LazyLoad Controller receives and processes the data and stores it in memory in the form of a Map. Once the Controller is restarted, all in-memory data is lost. When the ServiceFence is updated, the Status.MetricStatus of the servicefence is cleared and the dynamic service dependencies are lost.
Thinking
Step 1 Simple Persistence
The reason for the problem is that the cache map is initialised with a null value, but in fact, status.metricStatus of each ServiceFence is a good record of persistence dynamic service dependency and can be initialised by reversing servicefence.status.metricStatus -> cache map to complete the initialization.
The follow-up is still one-way cache map -> servicefence.status.metricStatus
Step 2 Time-sensitive Persistence
In Step 1, the contents of the cache map are never deleted. Suppose a dynamic dependency is not invoked again for a long time after a certain occurrence, at which point we should have a mechanism to delete the dynamic dependency.
One scenario would be to record when the accesslog metric is generated, set a global time threshold that triggers deletion, and delete the metric when the condition is met. we could even consider allowing users to specify a dedicated time threshold for a particular service.
One problem with this is that when there is a metric in the cache map, subsequent calls are direct and do not go through the global-sidecar, so that the same accesslog metric is not generated again. Wouldn't that mean that lazyload sees it as inactive, regardless of whether the call is active or not? Resulting in all metric being periodically deleted.
Design
Step 1 Simple Persistence
AccessLogConvertorConfig adds InitCache
When initializing the producer in lazyload, add the contents of the initialized initCache to get the ServiceFence information with the help of DynamicClient provided by bootstrap.Environment. You only need to convert these info to initCache later.
Step 2 Time-sensitive Persistence
TODO