About the result

[chaeyoun]

I trained the eve model with eve data, ran eval_codalab.py and got pkl file as a result. I also ran eval_codalabl.py and got pkl file from the pretrained model weights(from https://github.com/swook/EVE/releases/tag/v0.0 - eve_refinenet_CGRU_oa_skip.pt)
Then, I compared these two results and the numbers seem to match.
For example, from the pretrained model, I got [960. 540.] for PoG_px_final and got [963.0835 650.5635] for my model.

However, in the eve paper, table3 shows that the PoG_px in GRU model with oa+skip is 95.59
Numbers in paper is 1/10 of the numbers i got from eval_codalab and not sure what went wrong.
Are they supposed to match?
If they are not supposed to match, how do you calculate the numbers?

Also, in the result page of codalab, the gaze direction(angular error) is shown, but the eval_codalab.py doesn't store gaze direction. (Keys_to_store=['left pupil size' , 'right pupil', 'pog__px_initial', 'pog_px_final', 'timestamp'])
How should I get gaze direction error in degree?

[swook]

Hi @chaeyoun, thanks for your interest in our research and in reproducing our results.

I am not fully sure what you mean in your inquiry, but I think I can clarify what the PoG_px_final field means. This field refers to the point-of-gaze in pixel values, in the screen-space which is defined to be 1920 x 1080.

The error values in Table 3 are of Euclidean distance errors. You can see the corresponding values (evaluated on a portion of the full test set) at https://competitions.codalab.org/competitions/28954#results to find consistency in the order of magnitude.

The gaze direction error is calculated via a Python script that is part of the Codalab EVE challenge. I recommend that you submit your .pkl file in order to see the results via Codalab. Regarding this code repository,

EVE/src/models/common.py

Line 129 in c02cbb7

def calculate_combined_gaze_direction(avg_origin, avg_PoG, head_rotation,

and https://github.com/swook/EVE/blob/master/src/models/eve.py#L437 might point you in the right direction.