Pex 552/on demand detection triggers

[pyth0n1c]

For integration testing, instead of scheduling a savedsearch to run and waiting a predefined amount of time, use the API to call the search right away. This can be a significant improvement in terms of the amount of time spent waiting for RBA/Notable artifacts to be generated.

[xqi-splunk]

Code Changes

• Use dispatch() to call the SavedSearch right away
• Update the wait for risk/notable while loop and retry logic
  • Based on experiments, 45 detections need to retry dispatch() or run dispatch() after 50s to 120s after the savedsearch is enabled. Therefore, add a retry logic after 90s (50s is the corner case, most time needed falls under two intervals 70-80, and 120-130)
  • Based on experiments, some detections need extra waiting time for the risk/notable objects to be generated (like 20+ seconds). But since the validation process itself would need around 5s - 10s, we just let the validation process work as the natural wait time. Extra exponential wait time is added after 30s to avoid non necessary wait time. If wait time is added at the first place, regardless of the 30s, it would add 20 minutes extra test execution time.

Test Outcome

• Decrease execute-testcases runtime from previous 111 minutes to 41 minutes.
• Test result remain consistent, only Potential password in username failed. (No touch in the validation logic of risk/notable in this MR)

[xqi-splunk]

Code Changes

• Update query for validate risk/notables to use latest job.sid after dispatch()

Test Outcome

• Test pipeline execute-testcases runtime 45 minutes
• Test result remain consistent, only Potential password in username failed.

[cmcginley-splunk]

Looks great Xiaonan! And the performance improvement is so so exciting; great job

Have a couple of structural changes requested, as well as sompoints for clarification

[cmcginley-splunk]

Looks great! A few more small tweaks and some points for conversation. @pyth0n1c please weigh in where appropriate :)

[cmcginley-splunk]

For option 2, I was only suggesting that bool field in the case we still wanted to do cleanup pre-test. But I agree with Eric for all the reasons mentioned. I would remove this code path (or throw on self.sid is None) and remove the pre-test cleanup

[cmcginley-splunk]

See my comment above

[cmcginley-splunk]

Looks great xiaonan :) approved