status is completed before WMAS test results are published

[shilpa-gaikwad]

I use REST API "api/results/{token}/compact" to get results after status of session is completed. But it seems that response to that API is not what actual results are. I get right results few seconds or minutes after WMAS session is completed/finished.
e.g. If WMAS session is finished for test group "websockets" and I make GET request to "api/results/{token}/compact" immediately after session is ended, I get results as {'websockets': {'pass': 223, 'fail': 0, 'timeout': 1, 'not_run': 0, 'total': 42, 'complete': 29}}
And if I make GET request to same API with same session after 10 seconds, then results are: {'websockets': {'pass': 223, 'fail': 0, 'timeout': 6, 'not_run': 0, 'total': 42, 'complete': 34}}
It takes almost a minute or so to get actual results after session is ended. There is lot of delay in between actual results and session completion time.

I think session's status should not be "completed" until final results are ready.

Same is the case when user visits https://webapitests2019.ctawave.org/wave/results.html?token=token page. It does not display right results for at least approximately 1 minute in "API Results" section (observed this when websockets tests are run).

[louaybassbouss]

@shilpa-gaikwad the session complete status is related to the test run and not when the results are available. @FritzHeiden can you please have a look on it and explain the internal behaviour? @shilpa-gaikwad we will keep you updated.

[shilpa-gaikwad]

Thanks @louaybassbouss
I would like to know, the correct way of retrieving accurate final results (compact results) using REST API after session is ended/completed/finished.

[louaybassbouss]

@FritzHeiden will provide more details by tomorrow

[FritzHeiden]

@shilpa-gaikwad Thanks for bringing this up. The reason for this behavior is, as @louaybassbouss already hinted at, due to the fact that the complete status is related to test execution. So when a test moves on to the next test without posting the results to the server, the session may complete before the internal timeout of the server marks this test as timed out.

I agree that the resulting behavior is confusing and not as expected. We will change the behavior to mark all pending tests as timed out as soon as the session is completed.

For now you can assume that the pass and fail count won't change after the session is complete, and that there are as many timeout results missing as the difference between complete and total.

[louaybassbouss]

@JohnRiv FYI @FritzHeiden is working on it and we will have a fix for it very soon. We will apply the fix to all WMAS versions. For WMAS 2018, we will wait after #78 is fixed (latest version of the WPT test runner is integrated)

[FritzHeiden]

#95 contains a fix for this. First tests look promising.

[louaybassbouss]

Thanks @FritzHeiden #95 is now merged and applied to all WMAS versions. New GH tags are created and the versions table is updated accordingly. This issue can be closed

[JohnRiv]

Closing as #95 fixes it