Result with same feature in input concatenates instead of merging

[gpucciani]

Hello,

We have an issue when aggregating results of scenarios that run in parallel.
In our example (attached screenshots) we used two json with the same feature, and we would expect the script to do an "intelligent" merge. That would result in a json with only one feature. It seems however that the script concatenates only the two files.
Are we doing something wrong? Can you please clarify a bit more the type of merge we can expect and if there are some requirements on the input files?

I see that in "cucumber-json-merge/test/fixtures/" files 1 and 2 are related to different features.

Thank you!

result

file1:

file2:

Thanks a lot for your help!

[ljunggren]

This is exactly what I see as well. What we have seen from the Cucumber report Jenkins plugin (https://github.com/damianszczepanik/cucumber-reporting) is that the built-in merge manages to merge these based on id (here: "m125").

From our point of view, the ideal result would be that both "Scenarios" would be children of the common feature "Order States".

[bitcoder]

the current merge is very basic: its purpose is to merge cucumber.json reports by some tools that generate these, one per each feature.
It doesn't handle scenarios as the one detailed on this issue, where one example runs on one machine and another in other machine, which I wasn't even aware that they were possible.
A PR is welcome though :)

[bitcoder]

@gpucciani how do you run your scenario outlines so that each example runs in a different host? do you have a sample project code you can share?

[ljunggren]

Maybe I can shed some light on this. Boozang has a built-in dispatcher that can support multiple workers (in this case Docker containers running Puppeteer and Xvfb). The dispatcher supports dividing up the scenario iterations between different workers in the "Scenario Outline" to guarantee even workload.

I have attached two reports generated from two different workers running two different scenario iterations within the same "Scenario Outline". The ideal case would be to be able to merge these so that XRay will display them together.

report_cucumber-m125-t1-0.json.txt
report_cucumber-m125-t1-1.json.txt

[bitcoder]

Note: in cucumber (ruby and JVM), multiple elements related to the examples have a dynamic id such as:

as-a-user--i-can-calculate-the-sum-of-two-numbers;sum-of-two-positive-numbers;;2
as-a-user--i-can-calculate-the-sum-of-two-numbers;sum-of-two-positive-numbers;;3
...

therefore it seems that thesse id's are not being used as cucumber official tools do, which is an additional challenge.

[ljunggren]

If I understand it correctly the key is to have dynamic id:s for the different data iterations in the Scenario outline. We can totally do this - in fact we have full control over the format of the Cucumber reports.

Do you think you can find an example using two different feature files containing scenarios from the same scenario outline that can be successfully merged into one? If so, we can simply follow that example format and make it work that way.

[gpucciani]

@bitcoder Is it possible to have such an example as mentioned by @ljunggren ?

[bitcoder]

Hi,
any simple cucumber project based on cucumber official/upstream tools can generate such a report; I've provided a ruby example, where data.json contains the output of running.
Please note, that even if your tool generates cucumber JSON report with this syntax, the current merge utility in this project is unable to handle it.. that logic is missing and it may be tricky to implement.

cucumber-ruby-calc.zip

[ljunggren]

Hi Sergio,

We've created an experimental fork on cucumber-json-merge that supports this case.

https://github.com/ljunggren/cucumber-json-merge-multiworker

I hope that is ok by you! It's still experimental so not sure how well it works yet.

Let us know if something doesn't look right to you

Best Regards,
Mats

[bitcoder]

Hi @ljunggren ,
I'm ok with it, if the implementation is fine, we can perhaps merge it in the existing repo.
I looked briefly at the code but I'm unsure it deals well with the merge operation.
We need to handle these cases, at least:

1. cucumber json reports related to different Features; merging them is straighforward (current implement in this project) as it is appending the first object in each JSON report
2. cucumber json reports eventually related to the same Feature, where some json reports may contain results for specific iterations in eventual scenario outlines that may exist there; merging this should have in mind the id of the corresponding feature, and it should guarantee the proper order of the elements related to each entry of the examples section. This order is used by Xray to know exactly to which iteration (i.e. example entry) it corresponds to; other tools may use the same logic as there is no explicit way to indicate that a given element corresponds to a given example entry.
3. a mix of the previous

Regards
Sérgio

[ljunggren]

Hi Sérgio,

Great! As you say, we are not yet sure it deals with all scenarios you describe. Let us confirm that it works for the case we have seen in TCS, and take it from there.

As soon as it's confirmed I think the best approach would be to add the "merge scenarios" you describe to "test/fixtures" in your repo. That way I can make sure it works for those scenarios and create unit tests for it, then you can merge it back when you feel confident it works.

Best Regards,
Mats