PEX-653: KVStore Versioning Validation

[xqi-splunk]

Overview

This PR make below changes:

1. Add kvstore versioning validation while preserving previous datastore based versioning validation logic
2. Add new wait versioning activation logic for kvstore since in ES 8.3.0 versioning is activated by default but would need to wait until is fully activated. Using new API endpoint to check versioning activation.
3. Create a CMSEvent class for cms_event

Testing

1. Tested with ES 8.3.0:

2. Tested with ES 8.0.2:

Note: The reason doesn't test with ES version between 8.0.4 and 8.2.3 is there is known issue with version validation in these versions

[xqi-splunk]

Confirmed with David that the message only populated when there is error occurred. Therefore, adding logic check if message presents:

                            # If there is error message versioning is not activated properly
                            if "message" in app:
                                return False

Also check if the ESCU being installed is the same version as the one being tested:

                            # If the installed verion is not the same as the test version
                            if app.get("version") != self.global_config.app.version:
                                return False

Pass local testing:

[xqi-splunk]

The ESCU testing using this feature branch + the ES 8.3.0 build (splunk_app_es-8.3.0-211918.spl) + CIM 6.3.0 passed:

31037530

[cmcginley-splunk]

Is this step needed? kvstore versioning should be activated by default; I don't think we need to wait for it to activate

[xqi-splunk]

According to David, although the kvstore versioning is activated by default, it takes some time for it to be ready. So I added the wait logic here, just to ensure that it's activated successfully (like some time for versioning to setup properly) before searching in kvstore.
From the testing pipeline, the setup time is usually very quick.

[cmcginley-splunk]

That sounds like we're waiting for versioning initialization, not activation, no? Possible I'm not understanding something in the new model, but previously versioning activation was instantaneous, and we were just waiting for the parser to ingest everything

[xqi-splunk]

Tested with ES 8.0.4 and ES 8.3.0:

1. ES 8.0.4: 31452104
2. ES 8.3.0: 31451968

[cmcginley-splunk]

Approve. We will follow up on the open thread later