Description

This project reproduces the results from the paper Extracting Commonsense Properties from Embeddings with Limited Human Guidance. The final summary of this project is summarized in this report.

Contents

• data/: Contains data files
• figs/: Contains output plots, also added in the report
• output_dumps/: The results of all runs for reference

Code Walkthrough

Scripts

Main Script: script.py

Argument Description
--data data/ : The folder containng the data files
--test verb_physics_test_5 : Test Data File
--devtest : {"true", "false"}, Whether to evaluate on dev data or not
--dev : Dev Data File
--train verb_physics_train_5 : Training Data File
--embtype lstm : {word2vec, glove, lstm} 
--embeddingSize 1024 : Embedding size; give 1024 for lstm and 300 for other two
--test_relation all : {big, heavy, rigid, strong, fast, all} These are the options for verb physics
--zero False : {"true", "false"}, Whether you want zero shot learning on the given test_relation
--reverse true : {"true", "false"}, Whether you want to consider reverse tuples while evaluation
--poleSensitivity False : {"true", "false"}: If you want to test on poleWord1 and poleWord2.
--poleWord1 slow : Relation for R<
--poleWord2 speedy : Relation for R>
--onepole : {"true", "false"}: To use PCE(one-pole)
--four : {"true", "false"}: To use 4 way model

# To Run the script with default configuration
python script.py

# To Run one configuration of 3 way model(Example Configuration)
python script.py --train verb_physics_train_5 --dev verb_physics_dev_5 --test verb_physics_test_5 --devtest true --four false --reverse true --onepole false

# To Run one configuration of 4 way model(Example Configuration)
python script.py --train train_data --test test_data --devtest false --four true --reverse true --onepole false

Other Scripts:

datautil.py: Contains functions for creating dictionary, preprocessing and filtering data

models.py: Contains Emb_Similarity and Model(the main one)

train_evaluate.py: Train, Evaluate and active learning utilities(find_LC_query, find_EMC_query, find_uncertain_synthesis_query)

Embedding Similarity Baseline

python emb_similarity.py

Majority Baseline

python majority.py

Active Learning Experiments

Runs 4 active approaches(Random, LC, EMC and Synthesis Based) and Plots the training examples vs Accuracy Plot. For details, please refer the report. python active_learning.py

Score Analysis

Trains the model on verb physics dataset usign same parameters and calculates a proxy score of relatedness and plots the relative size and weights of some objects python score_analysis.py

Shell script that runs all configurations in Table1 10 times

sh run_table1.sh > table1_result sh run_table2.sh > table2_result sh run_table3.sh > table3_result

Shell scripts for statistics and plots

Python script uses the results from the table1_result, table1_result and table3_result files, calculates mean, std and plots and save figures in figs folder python stats_table1.py python stats_table2.py python stats_table3.py

Significance Test File

python mcnemar.py