[BUG] Security config updates takes long time in clusters with Optimized PE and having large number of roles #5983
Description
What is the bug?
With #3870, much of the work of privilege evaluation is now done upfront by creating optimized data structures on node boostrap or if an index is created/deleted (change to cluster state)
An issue is observed when CRUD operation is performed on security roles/roles mapping on a cluster with large num of roles with optimized privelege evaluation enabled, it can take upto 5mins for the update. This results in a very degraded performance
Initial analysis
This behaviour seems similar to #5464 where we observed node initialization was taking long time. In this case when security config updates is performed, the precomputed privilege evaluation (PE) cache is updated synchronously. If the cluster has large num of roles and roles mapping, these updates to precomputed PE cache takes longer time