Improve the precision of the reward signal we're using for ground truth

[neubig]

@adityasoni9998 has an analysis of some places where the reward signal is failing and we could improve those.

[adityasoni9998]

I have found the following edge cases for which the ground truth parser behaves incorrectly:

• Added files/classes/functions won’t work. The names written by a developer and by an agent won’t match.
• Changing attributes/variables inside a class: Detects at file-level correctly, produces empty change list for entity and module-level
• Global-level changes (like global variables) and imports: detects at file-level correctly, produces empty change-list for entity-level and module-level
• AST-level parsing issues: changes like adding @wrapper on top of a function, change in inheritance of a class (class X <--> class X(Y)), etc. are only detected at file-level, not at module/entity level. If the file contains some hard-coded characters which AST fails to parse, then in that case as well the list of entity/module-level changes is empty
• False-positives: even if the file just added some new-lines, comments etc. these are added to ground truth (possibly because their AST-level nodes differ).

[adityasoni9998]

Hi Graham sir! I have tried some of the corner cases here:

• Changing attributes/variables inside a class and change in inheritance of a class (class X <--> class X(Y)) are now detected at a module-level (previously this was only file-level)
• Parsing the function-level docstrings which was considered as a change initially is now ignored. So changes to function-level docstrings are ignored (previously only class-level docstrings were ignored)
• New functions added to an existent class: previously if a new function was added to an existent class, this change was not detected at a module-level but now it is detected.

Fixing AST-level issues on our side is very tricky and other cases are probably not that important for us: global variables only need to be detected at file-level so we should be ok with empty class/function-level changes here. Also the PRs for which new files were added are hard to use in training because a code-search agent cannot really predict the name/location of a non-existent file. But new functions are detected at a class/file-level and new classes are detected at a file-level in our logic.

My code is available here