fail/skip/error extraction, score calculation and Topfails view

[jjchen]

• Implement needed backend functionality to return the list of worst failures based on internal data structures in the CassandraBackend.
  • extract fails, skips, errors from result XML
  • calculate a score and insert it into test_by_score
• design a back-end interface for top failure retrieval and implement it (query on top fails and/or in-memory queue)
• hook up the front-end to display this list.

Current interface: http://parsoid.wmflabs.org:8001/topfails

[choucalate]

Hi! If you'd like I could work on this as well

[jjchen]

Sure, thoughts on division of tasks?

[choucalate]

Sebastian has already done the first part of the first one - extracting fails, skips ,and errors https://github.com/gwicke/testreduce/pull/15/files in server.js
We have to probably do the backend insertion in CassandraBackend, but I'm not sure why he commented out his old one?

For designing a back-end interface for top failure retrieval- I'm not sure how this one works... is it for a particular commit?

Hooking up the front-end - I'm going to do a mock of this in a couple days, so we can probably work off of that

[choucalate]

Hi jjchen, I was thinking a bit on the implementation of in-memory queue AND a query on top fails

Basically if the server is just starting up, the only option is to query on top fails
I was thinking that maybe we could just use test_by_score table, but there's no way for us to do a score ascending or score descending query as of right now unless we make it part of the PRIMARY_KEY(.... , score,...)
or
we could simply bring everything into memory and compute it, but I think that might be really expensive, given 150k tests (per commit) * hundreds of commits

As for the in-memory queue, we'd probably have the same as the current testQueue and add data every time we save into test_by_score or do a query

What are your thoughts on this so far?

[jjchen]

Did the code that was removed from addResult by https://github.com/gwicke/testreduce/pull/15/files get replaced/moved (wasn't that doing the extraction, calculation of the score, and insertion into test_by_score)? Also, by "in-memory queue AND a query on top fails" do you mean keeping both a top_fails table and a top_fails in-memory queue? So we would read from the top fails table into the queue when the server starts up, then update both whenever we save into test_by_score?

[choucalate]

Yah I remember it being there in a previous pull request, I think it's still being fixed atm

Mm.. as for your second point, "a top_fails table and a top_fails in-memory queue" - I guess that could work, but I was just thinking of making use of the test_by_score table and selecting the tests that have the most fails

But I'm not sure how a top_fails table would work since we would probably have to do an updating of the tests instead of simple insertions, which correct me if I'm wrong, isn't the best in Cassandra?

[jjchen]

Is someone working on it right now? If not, I could do that.

Okay, yeah, that's what I was originally thinking too (using the test_by_score table), just wasn't sure if "query on top fails" was meant as an alternative to it. I guess I'm not exactly sure what 'top fails' including all commits actually means though?

[jjchen]

Hi @choucalate ,
I've added code for extracting from result XML, and calculating a score and inserting it into test_by_score back, and started some stuff for the in-memory queue, which would read from test_by_score when the server starts up and adds data every time we save into test_by_score. For the start up query on top fails, I'm using the second option you mentioned (just bringing everything into memory and computing it) - as you said, this could be really expensive, but I wasn't sure how else to do it - even if we add score to the primary key of the test_by_score we'll still need the commit and test to do a score descending query, right?

[brando90]

Hi @jjchen,

Where have you added that code for extracting XML results?

Can I get it by git fetch upstream?

[brando90]

@choucalate @jjchen I read a little bit of the discussion about the score. Why don't you make it a secondary index instead of making it the primary key? Wouldn't help query it faster?

[choucalate]

woops closing it was an accident lol
yah if we add score to the primary key, then we'll need the commit and test, I didn't think about that. But I think your idea for adding to the in-memory queue every time we do something to test_by_score should be fine.

As for what brando was saying, I'm not sure if there will be any side effects, but that might actually make the querying easier.

do you want to split up the 2 tasks?

1. adding it to the in-memory queue in addResult?
2. Computing everything in start up?

After that, passing it to the front end should be pretty simple