generates empty list error

[vinayakumarr]

Due to memory problem i have used only 1000 files in train- pos and neg and when i execute python src/train.py, after some time it generate te following eror. How to solve this. Please see the attached file

[rcampos2029]

In the original code, we performed 40 batches with 600 files in each (300 pos/300 neg). So your code fails on batch 2 because it runs out of files to train on. I just updated the code to have variables to tune the development set size, number of batches, and batch size on line 46. Change these variables so that DEV_SIZE + BATCH_SIZE*BATCHES = len(train)

[vinayakumarr]

500+(50*10)=len(train). Successfully works. test data set is 250. for all test data set it is classifying as 0. How to see the fscore and precision and accuracy. I came to know in tmp folder batch files will be create and that has these information. But all files is showing accuracy precision and recall as 1. Do you where the problem has occurred or whether the results are correct.

[rcampos2029]

that is odd that they all show up as 1. What does it say for TP, TN, FP, and FN? predict.py has the code for the metrics.

[vinayakumarr]

FP.txt and FN.txt files doesn't contain anything. But TP.txt and TN.txt contains 1001 lines (includes the name of files). You can see a part of TP.txt and TN.txt in the below attached image

TP

TN

[vinayakumarr]

predict.py shows below result

pos (train) contains 1002 files and neg (train) contains 1002 and test contains 252 files

[rcampos2029]

Are the TN and TP pictures right? TP should display positive, and TN should show negative. It's also odd that it lists 2002 files instead of 2004 if there are 1002 files in each directory. Are there any batch_#.txt files in your tmp directory (such as batch_0.txt?

[vinayakumarr]

i mentioned wrong in previous post, such that pos (train) contains 1001 files and neg (train) contains 1001 and test contains 251 files

inside src/tmp folder batch_*.txt contains 50 files

each file has precision, recall, f1 and accuracy. it show total test files as 1000 but test has only 251 files.

what these batch files are?

How to correct this?

[rcampos2029]

So the batch files are calculating the metrics on the development set during each batch iteration.
The reason it says 1000 files is because your dev set was specified as 500, and there was a bug that took 500 from positive and negative instead of 500 total.
I just updated the code to fix this, so now if you specify a dev size of 500, it will actually be 500.

Running predict by itself will do the calculations on the entire training set. To classify the test set, run test.py. This will generate a csv file of ID,LABEL.

[rcampos2029]

Oops, just completed the push now. Try pulling, changing the SIZE variables and running again.

[vinayakumarr]

I have 1000 train/pos and 1000/neg, so i used

DEV_SIZE = 1000 # Split evenly between pos/neg
BATCHES = 20
BATCH_SIZE = 50

is it correct?

[rcampos2029]

Yes.

[vinayakumarr]

now again i get the previously solved error

[rcampos2029]

That's odd, on line 161 of train.py, add the following just to make sure the script see 2000 files:
print len(training_set) ; exit()

Could you show batch_0.txt?

[vinayakumarr]

shows 1998

[rcampos2029]

That would explain the pop from empty list error, and that means train.py only sees 999 files in each directory (pos/neg).