Train and Test pkl seems to have missing added and removed code

[Manas-Embold]

import pickle
data = pickle.load(open('/content/openstack_train.pkl', 'rb'))
ids, labels, msgs, codes = data

If we look at codes, it does't contain actual added and removed code

[['added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code'],
['added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code',
'added _ code removed _ code'],
['added _ code removed _ code', 'added _ code removed _ code'],

[Manas-Embold]

Predictions run and is still able to produce AUC Score of 74

100% 21/21 [00:01<00:00, 11.19it/s]
Test data -- AUC score: 0.7486763593579291

How is it happening without data?

[Manas-Embold]

If i try to run the same code on pkl from jit task of cc2vec which contains added code and removed code i get following error.
Link to data for jit task of cc2vec: https://zenodo.org/record/3965149#.X2VeP5MzY1J
pkl file: data/jit/openstack_train.pkl

Error:
Traceback (most recent call last):
File "main.py", line 58, in
pad_code = padding_data(data=codes, dictionary=dict_code, params=params, type='code')
File "/content/padding.py", line 36, in padding_data
pad_code = padding_commit_code(data=data, max_line=params.code_line, max_length=params.code_length)
File "/content/padding.py", line 61, in padding_commit_code
padding_length = padding_commit_code_length(data=data, max_length=max_length)
File "/content/padding.py", line 66, in padding_commit_code_length
return [padding_multiple_length(lines=commit, max_length=max_length) for commit in data]
File "/content/padding.py", line 66, in
return [padding_multiple_length(lines=commit, max_length=max_length) for commit in data]
File "/content/padding.py", line 18, in padding_multiple_length
return [padding_length(line=l, max_length=max_length) for l in lines]
File "/content/padding.py", line 18, in
return [padding_length(line=l, max_length=max_length) for l in lines]
File "/content/padding.py", line 21, in padding_length
line_length = len(line.split())
AttributeError: 'dict' object has no attribute 'split'

[hvdthong]

Many thanks for your comments. I think I uploaded the wrong dataset. Let me try to find the correct one and I will let you know. Thank you and have a great day.

[Manas-Embold]

That would be super helpful.
Many Thanks.

[Manas-Embold]

Hi Thong,

Did you get a chance to find the correct dataset.
Or can you guide me how to convert the openstack_train.pkl to openstack_train_DExtended.pkl

I would be thankful for the same.

[hvdthong]

Hi Manas,

I'm a bit busy now as I have some deadlines which are coming. Moreover, because of COVID-19, I have to work at home. I will find the time to get the data. At this moment, I don't know how to convert the openstack_train.pkl to openstack_train_DExtended.pkl as their formats are different. Thank you and have a great day.

[jiojio718]

有正确数据集吗？