Run our own model using EMTk

[bcdasilv]

Instead of using the pre-built EMTk model which was trained over StackOverflow discussion, we should train and test a new model based on the dataset we're using (code review comments from SentiCR).

Use the oracle.xlsx dataset to train a new model in EMTk using 10-fold cross validation.

Report the performance of this new model in a wikipage in our repo.

[steven19992017]

when attempting to train my own emotion model, i ran into the follow errors:

Additionally, I don't really understand instructions on formatting the file for training the new model.
id;label;text
…
22;NO;"""Excellent! This is exactly what I needed. Thanks!"""
23;YES;"""FEAR!!!!!!!!!!!"""
…
Also, I don't see how we could convert our polarity score from oracle.xlsx to emotions or "Yes" and "No" as mentioned on the example.

[steven19992017]

hours spent: 8 hours

[bcdasilv]

Thanks for the update.

First, it seems that you missed the '-p' parameter. See that issue they've reported: collab-uniba/EMTk#3

Also, make sure a file 'sample.csv' is available. Is that the sample file provided by them?
Are you trying to test-drive first with their sample file? See that section on their readme:
"Users can test-drive the emotion classification module by using the file /emotions_sample.csv, which contains only a handful of documents. Other more complex sample datasets are available at /emotions/java/DatasetSO/StackOverflowCSV."

Regarding the file format, I don't know exactly what you don't understand. The columns ID and text are straightforward. They mean the unique id of the text/row and the text is the actual content to be classified. However, the column 'label' does seem confusing to me. Maybe they're referring to the emotion label, although YES and NO don't seem to be emotion labels. They could be sentiment polarity labels (NO - non-negative, YES - negative) which they used as an example. So, if that's the same confusion you've seen, please open their csv files and check if there are other labels than just YES and NO. Also, hopefully, after fixing the issues above you can test-drive the emotion module and see the results to better understand the label column. Does it make sense?

Please, post updates with further questions, comments, or solutions. Thanks!

[steven19992017]

For some reason running docker desktop crashed on windows. I will look into the issue
,,

[bcdasilv]

That sucks. Any updates, @steven19992017 ?

[steven19992017]

I spent some time reinstalling docker as well as pulling the emtk image. I explored the example training files for the emtk model but I still find the labels of "Yes" and "No" confusing. Maybe they are for indicating whether or not they are gold standard datas? Besides, I don't see how just giving them sentences without classifying the actual emotions would train our new model. Based on my understanding, to train a model we need to tell them the correct emotion but they never mentioned anything about specifying our own emotion inputs.

[steven19992017]

Time log: 8 hours

[bcdasilv]

You're correct. In order to train, we would need a column to tell the correct emotion. The code review dataset we have (oracle.xls file) was labeled based on two-level sentiments (negative and non-negative). Let's talk more about it in our next meeting.