i_1006 Remember scoreboard type in ContestInformation#1146
Open
johnbrvc wants to merge 25 commits intopc2ccs:i1006_point_scoringfrom
Open
i_1006 Remember scoreboard type in ContestInformation#1146johnbrvc wants to merge 25 commits intopc2ccs:i1006_point_scoringfrom
johnbrvc wants to merge 25 commits intopc2ccs:i1006_point_scoringfrom
Conversation
Add basic support data structures. Add test unit for new classes.
Changes needed to be made since ContestImportUtilities was added and the common routines moved out of ContestSnakeYAMLLoader.
Add more JUnit tests to read an entire hierarchy. Add more test data containing a full example of a hierarchy of test data groups. Add group to TestCaseInfo.
Commented out the printing of the flat list of test cases in the TestDataGroupTest JUnit.
Test Data for CLICS problems are no longer read in directly in ContestSnakeYamlLoader, rather, a TestDataGroup object is created representing the hiearchy of test data, along with any present testdata.yaml files. ProblemDataFiles now has a list of TestDataGroup's (one per test case). Modified GUI of the problem edit, Test Data Files tab to show the Data Group each data test case is part of.
Create new package: edu.csus.ecs.pc2.graders. Created LegacyGrader class. Created JUnits to test grader class (need more tests).
JUnit found a bug with the way submissions are counted. Basically, rejects count when using "avg" as the scoring mode, and so their score values.
The JUnit tests for the ContestSnakeYamlLoader made some assumptions about the locations of the secret data. This is now handled automatically by the TestDataGroup class. Fix the routine in ContestSnakeYamlLoader to not put the "secret" folder on the directory path for data. Fix spelling error in JUnit print message. Make sure the static log is created before running junits since the TestDataGroup stuff will log things.
Some JUnit tests read a CDP that contains problem datafiles. These JUnits now make use of the newer TestDataGroups() class to read test data as a result of using ContestSnakeYamlLoader. This class requires a valid Log object. ContestSnakeYamlLoader uses StaticLog(), as a result, the static log must have been previously intialized. AbstractTestCase.ensureStaticLog() does this.
Forgot to add an ensureStaticLog(). Also added missing call to base class setupUp() in LoadICPCDataTest.
Note: currently the "scoreboard_type" is set to "pass-fail", because the ContestSnakeYAMLLoader does not recognize "scoring" as a legitimate value. This needs to be updated once ContestSnakeYAMLLoader is updated to recognize "scoring".
…changes i_1006 Adjust JUnits for new TestDataGroups
…sample_contest i1006: add "pointscoring" sample contest (v.1)
Allow "score" scoreboard type in addition to the standard "pass-fail". Save the scoreboard type in the contest's ContestInformation. Update CLICS contest API endpoint to use the scoreboard type defined in ContestInformation instead of hard-coding pass-fail. CI: ContestInformation was missing compares for many members in the isSameAs() method, so if any of these were different, isSameAs() would not detect that. CI: Add string constants for contest scoreboard type.
Some JUnit tests read a CDP that contains problem datafiles. These JUnits now make use of the newer TestDataGroups() class to read test data as a result of using ContestSnakeYamlLoader. This class requires a valid Log object. ContestSnakeYamlLoader uses StaticLog(), as a result, the static log must have been previously intialized. AbstractTestCase.ensureStaticLog() does this.
Changes to DSA to support both pass-fail and point scoring contests. Update CLICS API to deal with scoring contests and only put out scoring related properties if it is a point scoring contest.
clevengr
requested changes
Jul 16, 2025
Contributor
clevengr
left a comment
clevengr
left a comment
There was a problem hiding this comment.
I have reviewed all but one of the (20) changed files in this PR (the changes to DefaultScoringAlgorithm are extensive and I need more time to look at those).
I noted once place where I think there is a logic error (in DefaultPointScoringStandingsRecordComparator.java). Other than that I made only minor comments, which I'd like to see considered and addressed, but wouldn't stop me from Approving the PR.
Given that there's no practical way to TEST this PR until the entire Point-Scoring pipeline implementation is done, I would simply approve the PR once the comments I've made are addressed.
src/edu/csus/ecs/pc2/core/scoring/DefaultPointScoringStandingsRecordComparator.java
Outdated
Show resolved
Hide resolved
src/edu/csus/ecs/pc2/core/scoring/DefaultPointScoringStandingsRecordComparator.java
Outdated
Show resolved
Hide resolved
Fix spelling errors in comments. Fix wrong types in comments.
It's not intuitively obvious how the comparator arrives at its return result. Hopefully the comment added explains it.
clevengr
approved these changes
Jul 16, 2025
Contributor
clevengr
left a comment
clevengr
left a comment
There was a problem hiding this comment.
I have re-reviewed all changed files, and also completed a cursory review of DefaultScoringAlgorithm. I'm not sure about all the changes in DefaultScoringAlgorithm; they look reasonable but there are a lot of details in the changes. That said, and given that there's no way to do a run-time test of the PR, I'm approving the PR so we can move forward.
Comment on lines
+306
to
+308
| if (!solved) { | ||
| score = 0.0; | ||
| } |
Contributor
There was a problem hiding this comment.
This appears to be making a hard-coded decision that if the problem is not solved then the score is 0.0. It seems like there could be situations where the Grader could have assigned "some points" (for example, due to having passed "some tests" but not all) even if the problem wasn't "entirely solved". Shouldn't the score assignment here be coming from some input somewhere (like, the JudgementRecord) rather than being hard-coded to be zero?
Use Double object instead of primative. This way we don't have to filter and can just use the "only include if non-null" annotation instead of all the extra rtfilter code. Fix copyright date in ProblemDataFiles.
If using the LegacyGrader internally (and not as an external process), the "String gradeTestCases(ArrayList<String> testCaseResults)" may be used directly. The bypasses the extra steps of saving the list of judgment/score tuples to a file and having the grader open and read that file. In addition, the gradTestCases method returns the net judgment string, eg. "AC 50" on success, and null on failure.
Specification illudes to how to implement ignore_sample. It relies on the fact that it may be used only at the root level and there must be 2 groups (sample and secret) at the root level. *sigh* What a hacky way to implement it, but - that's the spec. Add JUnits to test the flag - 3 cases.
validCDP used to form the paths to judge's data on the assumption thta everything was in either sample or secret. With test data groups, that is no longer the case, so change validateCDP to use the files from the ProblemDataFiles, which, is probably what it should have done in the first place.
johnbrvc
force-pushed
the
i1006_point_scoring
branch
from
7f776b4 to
ae6ff6b
Compare
Collaborator
Author
|
This PR has been merged already to facilitate further development. Both the i1006_point_scoring and i1006_update_prs branches have this code. This PR will be kept open, however, for future review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of what the PR does
Allow "score" scoreboard type in addition to the standard "pass-fail". Save the scoreboard type in the contest's
ContestInformation.Update CLICS contest API endpoint to use the scoreboard type defined in
ContestInformationinstead of hard-coding pass-fail.CI:
ContestInformationwas missing compares for many members in theisSameAs()method, so if any of these were different,isSameAs()would not detect that.CI: Add string constants for contest scoreboard type.
Issue which the PR addresses
Partial fix for #1006
Environment in which the PR was developed (OS,IDE, Java version, etc.)
Windows 11
Precise steps for testing the PR (i.e., how to demonstrate that it works correctly)
Start fresh (no
profileorprofiles.properties)Start a server loading the
samps/pointscoringcontestIt should load correctly (note that is uses
scoreboard_type: score)