Bug in ~/tfx/database.py

[malizad]

Hi,

I just find to use TFX to extract the defined features from my Twitter dataset. So I populated the user_label_assignments table with my labeled users (not all users are labeled of course since I want to use this software to label them). I also defined the label in conf.json file as label: {individual: 1, organization: 2}. Then I ran the TFX using the following command:

$ tfx ~/conf.json ~/path/to/tweet_dir

But I keep receiving the following error which I think is related to a bug in the code:

Traceback (most recent call last):
File "/usr/local/bin/tfx", line 63, in
extractor.FeatureExtractor(conf, args.tweet_dir)
File "/usr/local/lib/python2.7/dist-packages/tfx/extractor.py", line 25, in init
user_set = users.UserSet(conf, tweet_dir)
File "/usr/local/lib/python2.7/dist-packages/tfx/users.py", line 76, in init
self.user_ids[label] = self.connection.get_users_for_label(label_id)
File "/usr/local/lib/python2.7/dist-packages/tfx/database.py", line 226, in get_users_for_label
return self.profiles[label]
KeyError: 2

I think the problem arises from the fact that in ~/tfx/database.py, the __init__ method of the JSONFile class, assigns all users to label 0. I mean here:

label = 0
if label not in self.profiles:
      self.profiles[label] = {}
if user_id not in self.profiles[label]:
      self.tweets[user_id] = []
      self.profiles[label][user_id] = tweet_json['user']

so it raises an error when the get_users_for_label method get called, since get_users_for_label looks for 1 and 2 as keys, but self.profiles only contains 0 as its key.

[jmccorriston]

I think you may be confusing the different input methods. If you choose to use the JSONFile input method, your labels will need to be in the JSON files, not in a DB. You should only be using one input method here. In the other issue, I described the Text File input as well (the label was one of the columns there).

[malizad]

Thanks.
In case of using the JSONFile input method:

1. How and where should I define the labels? Should there be one JSON file per user or just one file containing all users?

2. How can I specify the path to labels file? The TFX command only accepts 2 arguments including conf.file and tweet_dir. The syntax is $tfx [-h] [--debug] conf.file tweet_dir. How should I let the tfx know where my labels file is/are?

[jmccorriston]

After reviewing the database.py module, I am now remembering that the JSONFile class was designed specifically only for classifying users, hence why it doesn't support labeling. Remember, the Humanizr package was not designed for training in the first place.

That being said, you should be able to use the TextFile class and .tsv input to use the TFX for training.

I apologize for misleading you with my previous comment!

[malizad]

I'd be grateful if you could update the readme file for TFX with some clear instructions so that one can know what steps to take for running it. The current version is quite misleading:

The problem is that the instruction says one should create 3 tables in a DB and put their information in the conf.file and then run the TFX by the following command:

$ tfx [--debug] [path to conf.file]

However, running the above command raises an error and asks for the path to tweet_dir as a required input for the tfx command (i.e. it says the correct way of running tfx is by $ tfx [-h] [--debug] [path to conf file] [path to tweet_dir]. However, when I created the tweet_dir and specified its path for the TFX, it seems that it runs the JSONFILE option, not the TextFile option, and fails to get the table from the DB. So there is clearly a bug here to fix.

In sum, right now, I have created inputs for both JSON and Text methods. Please kindly let me know a way that works. The $ tfx conf.file tweet_dir does not work.

[jmccorriston]

Line 69 in users.py sets the input method. This is not a bug with TFX. Again, this is designed to be a tool used specifically for labeling an unlabeled set. If you want to use the TFX for training a set, you will need to modify it to do so. I do apologize for the confusing documentation with the TFX, it's something that I forgot to update when I pushed this because I didn't consider this use case! I'll have to update it.

[malizad]

You have a whole documentation on how to use the tfx independently, but now you're saying that independent usage of tfx is not part of this project's intention. Sorry, but I really don't get it.

Anyhow, thanks for your time.

Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

-------- Original message --------
From: James McCorriston notifications@github.com
Date: 11/11/2015 10:10 AM (GMT-05:00)
To: networkdynamics/humanizr humanizr@noreply.github.com
Cc: Meysam Alizadeh malizad2@masonlive.gmu.edu
Subject: Re: [humanizr] Bug in ~/tfx/database.py (#3)

Line 69 in users.py sets the input method. This is not a bug with TFX. Again, this is designed to be a tool used specifically for labeling an unlabeled set. If you want to use the TFX for training a set, you will need to modify it to do so. I do apologize for the confusing documentation with the TFX, it's something that I forgot to update when I pushed this because I didn't consider this use case! I'll have to update it.

Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-155808117.

[jmccorriston]

The TFX was initially created as an internal tool. Humanizr depends on TFX and in order to release Humanizr publicly, we decided to release a copy of the TFX as well. If you look at the top-level README file, you'll notice that the TFX isn't mentioned. The README in TFX is left over from when the tool was used internally and as you mentioned, is out of date. An update is definitely needed for it. Perhaps that update should include an explicit statement explaining that to be used separately/for training, TFX might need to be modified!

[lfarcy]

I'm jumping into this discussion b/c it feels related to what I'm intending to do.

I'd like to test a few additional features and build a new model that might lead to better results, eventually, especially for predicting when a Twitter account is organizational.

To achieve this goal, I need a dataset and a toolkit to build a new model. I also need to setup a benchmark b/w models.

Hence, a few questions

1. Is there a way to get the list 20273 screen names mentioned in the research paper ? Can it be publicly released ? Or can anyone provide me or the community with the original training and test sets used to build the model enclosed within the repository ?
2. Can I build a new model from tfx and ml-classifier bundled in this repository ? Apparently, TFX would have to be modified. I had a look at the code and can make some changes. I'd be grateful if I could get a hint at what should be modified or added, globally speaking.

Thanks for this work and your reply.

[lfarcy]

@jmccorriston Any chance you can help me? Thx.

[davidjurgens]

Second author jumping in here to help. The first author (James) might be a bit too busy to respond.

I'm jumping into this discussion b/c it feels related to what I'm intending to do. I'd like to test a few additional features and build a new model that might lead to better results, eventually, especially for predicting when a Twitter account is organizational. To achieve this goal, I need a dataset and a toolkit to build a new model. I also need to setup a benchmark b/w models. Hence, a few questions 1. Is there a way to get the list 20273 screen names mentioned in the research paper ? Can it be publicly released ? Or can anyone provide me or the community with the original training and test sets used to build the model enclosed within the repository ? Yes! It doesn't look like I have write-access to this repository so I

can't push the data there directly but I've hosted it on my page: http://jurgens.people.si.umich.edu/data/humanizr-data.zip That should be all the user IDs but you'll still need to pull a fresh set of tweets for each (Twitter doesn't allow the redistribution of tweets directly).

1. Can I build a new model from tfx and ml-classifier bundled in this repository ? Apparently, TFX would have to be modified. I had a look at the code and can make some changes. I'd be grateful if I could get a hint at what should be modified or added, globally speaking.

Yes, I thinks the tfx and ml-classifier implementations in this repo are feature complete so you can definitely modify them as much as you need. The ml-classifier is a wrapper around an SVM and tfx does all the feature extraction. If these cause your problems, it's likely not too much effort to roll your own versions to re-implement those parts of the paper, as none of the features were particularly complex. Thanks, David

Thanks for this work and your reply

—

…

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7uLefdPpp7k8XKl4yuZyJUrNYAmhRMks5tI4W0gaJpZM4GfQCk> .

[lfarcy]

@davidjurgens Thanks for your reply! I was quite busy those last days. That's why I did not follow up before now.

I'm gonna download the dataset you made available on your web site and refresh the tweets through the Twitter API. I will also make stats about how many still exist and are still active on Twitter.

I also need to take time and take over the TFX and ml-classifier codebase. I'll let you know my progress.