Not rejudging deprecated languages or when merging test data on upload

[puqeko]

C++11 submissions are not being rejudged. Qless has no errors.

eg
https://train.nzoi.org.nz/submissions/47648

[technobird22]

Some sample cases are stuck as "pending".
eg. https://train.nzoi.org.nz/submissions/37968

[Holmes98]

Some sample cases are stuck as "pending".

This happens to existing submissions when the test data is updated using the "Merge with existing test cases" option. Existing submissions aren't rejudged automatically, so any new test cases/sets will show as "pending" for those submissions. New submissions shouldn't be affected.

[puqeko]

Hmmm. Some options re the merge upload...

1. Do nothing. Not useful but likely to happen.
2. Add a case which replaces "pending" with "outdated - please resubmit" or something more fitting. A user who passes the old test data continues to maintain their score but at-least the submission summary is a bit clearer that the changed test cases are not "pending" but instead the score is now outdated.
3. Trigger the same automatic rejudge as for the ordinary upload. Re-triggering the rejudging process more often could consume more server resources? @akampjes?
4. Re-evaluate only the updated/merged cases for each submission and update the judged outcome accordingly. This sounds like a lot of work when there are many other issues which require more urgent attention.
5. Remove the "Merge with existing test cases" option. Do we really need it? Yes, it would be slower to re-upload large test sets and potentially increase server data usage, but it would encourage getting the test data right first time, prevent this issue from getting worse, and reduce the amount of code that needs to be maintained.

For all options except option 2, past submissions with this problem will continue to incorrectly show "pending". So perhaps option 2 and option 5 together would offer the best compromise of practical and useful? Thoughts?

[Holmes98]

To clarify, the same thing happens with the "replace existing test cases" option, except that every test case gets stuck as "pending". So I'd rather keep the option to merge, which at least allows you to preserve some of the existing judging data (it's useful for adding sample test sets to old problems).

I'd be happy with (2) if anyone wants to add that. It doesn't seem trivial to implement though.

[akampjes]

(2) does seem the most clear, rather than attempting to handle updating test cases transparently and having to handle any additional cases / race conditions.