Q/A experiments report

[arezae]

A comparison report on Huggingface Q/A system fine-tune with our data.

1. Baseline model is Q/A model from Huggingface - bert_base
2. (1) + fine tune language model on our data
3. (1) + finetune Q/A on our 200 Q/A
4. (2) + fine tune on our 200 Q/A

We need to know how important is the fine-tune on our language model and fine-tune on our Q/A

[nafarieh]

1- The bert-base model which fine tuned by deepset/bert-base-cased-squad2 model:

09/19/2021 16:28:58 - INFO - main - Results: {'exact': 5.687203791469194, 'f1': 31.57154563090946, 'total': 211, 'HasAns_exact': 5.687203791469194, 'HasAns_f1': 31.57154563090946, 'HasAns_total': 211, 'best_exact': 5.687203791469194, 'best_exact_thresh': 0.0, 'best_f1': 31.57154563090946, 'best_f1_thresh': 0.0}

---

2- The bert-base model which fine tuned by MLM in our Dataset + QA Squad model:

09/19/2021 18:54:02 - INFO - main - Results: {'exact': 10.42654028436019, 'f1': 38.48925966485835, 'total': 211, 'HasAns_exact': 10.42654028436019, 'HasAns_f1': 38.48925966485835, 'HasAns_total': 211, 'best_exact': 10.42654028436019, 'best_exact_thresh': 0.0, 'best_f1': 38.48925966485835, 'best_f1_thresh': 0.0}

---

3- The deepset/bert-base-cased-squad2 model which fine tuned by QA Squad + QA Our Dataset model:

09/20/2021 05:08:18 - INFO - main - Results: {'exact': 47.39336492890995, 'f1': 71.79348969128544, 'total': 211, 'HasAns_exact': 47.39336492890995, 'HasAns_f1': 71.79348969128544, 'HasAns_total': 211, 'best_exact': 47.39336492890995, 'best_exact_thresh': 0.0, 'best_f1': 71.79348969128544, 'best_f1_thresh': 0.0}

---

4- The bert-base model which fine tuned by MLM in our Dataset + QA Squad + QA Our Dataset model:

09/19/2021 17:47:46 - INFO - main - Results: {'exact': 50.71090047393365, 'f1': 74.54551157490803, 'total': 211, 'HasAns_exact': 50.71090047393365, 'HasAns_f1': 74.54551157490803, 'HasAns_total': 211, 'best_exact': 50.71090047393365, 'best_exact_thresh': 0.0, 'best_f1': 74.54551157490803, 'best_f1_thresh': 0.0}

[nafarieh]

1- The squeezebert model which fine tuned by mrm8488/squeezebert-finetuned-squadv2 model:

09/19/2021 09:46:21 - INFO - main - Results: {'exact': 4.739336492890995, 'f1': 31.57909317121808, 'total': 211, 'HasAns_exact': 4.739336492890995, 'HasAns_f1': 31.57909317121808, 'HasAns_total': 211, 'best_exact': 4.739336492890995, 'best_exact_thresh': 0.0, 'best_f1': 31.57909317121808, 'best_f1_thresh': 0.0}

---

2- The squeezebert model which fine tuned by MLM in our Dataset + QA Squad model:

09/19/2021 13:39:31 - INFO - main - Results: {'exact': 3.7914691943127963, 'f1': 25.861355030166404, 'total': 211, 'HasAns_exact': 3.7914691943127963, 'HasAns_f1': 25.861355030166404, 'HasAns_total': 211, 'best_exact': 3.7914691943127963, 'best_exact_thresh': 0.0, 'best_f1': 25.861355030166404, 'best_f1_thresh': 0.0}

---

3- The mrm8488/squeezebert-finetuned-squadv2 model which fine tuned by QA Squad + QA Our Dataset model:

09/19/2021 13:59:24 - INFO - main - Results: {'exact': 36.018957345971565, 'f1': 65.08729223891349, 'total': 211, 'HasAns_exact': 36.018957345971565, 'HasAns_f1': 65.08729223891349, 'HasAns_total': 211, 'best_exact': 36.018957345971565, 'best_exact_thresh': 0.0, 'best_f1': 65.08729223891349, 'best_f1_thresh': 0.0}

---

4- The squeezebert model which fine tuned by MLM in our Dataset + QA Squad + QA Our Dataset model:

09/19/2021 11:48:06 - INFO - main - Results: {'exact': 40.758293838862556, 'f1': 67.35364641337732, 'total': 211, 'HasAns_exact': 40.758293838862556, 'HasAns_f1': 67.35364641337732, 'HasAns_total': 211, 'best_exact': 40.758293838862556, 'best_exact_thresh': 0.0, 'best_f1': 67.35364641337732, 'best_f1_thresh': 0.0}

[arezae]

Thanks for the updates. It would easier to compare if have them in a single table or graph.

[Ali-Razmdideh]

Thanks for your note.

model	exact	f1	total	HasAns_exact	HasAns_f1	HasAns_total	best_exact	best_exact_thresh	best_f1	best_f1_thresh
Bert + Squad	5.687203791469194	31.57154563090946	211	5.687203791469194	31.57154563090946	211	5.687203791469194	0.0	31.57154563090946	0.0
FineTuned Bert + Squad	10.42654028436019	38.48925966485835	211	10.42654028436019	38.48925966485835	211	10.42654028436019	0.0	38.48925966485835	0.0
Bert + FineTuned QA	47.39336492890995	71.79348969128544	211	47.39336492890995	71.79348969128544	211	47.39336492890995	0.0	71.79348969128544	0.0
FineTuned Bert + FineTuned QA	50.71090047393365	74.54551157490803	211	50.71090047393365	74.54551157490803	211	50.71090047393365	0.0	74.54551157490803	0.0
SqueezeBERT + Squad	4.739336492890995	31.57909317121808	211	4.739336492890995	31.57909317121808	211	4.739336492890995	0.0	31.57909317121808	0.0
FineTuned SqueezeBERT + Squad	3.7914691943127963	25.861355030166404	211	3.7914691943127963	25.861355030166404	211	3.7914691943127963	0.0	25.861355030166404	0.0
SqueezeBERT + FineTuned Squad	36.018957345971565	65.08729223891349	211	36.018957345971565	65.08729223891349	211	47.39336492890995	0.0	65.08729223891349	0.0
FineTuned SqueezeBERT + FineTuned QA	40.758293838862556,	67.35364641337732	211	40.758293838862556	67.35364641337732	211	40.758293838862556	0.0	67.35364641337732	0.0

[nafarieh]

Thanks for your note.

model	exact	f1	total	HasAns_exact	HasAns_f1	HasAns_total	best_exact	best_exact_thresh	best_f1	best_f1_thresh
Baseline Q/A bert_base	5.687203791469194	31.57154563090946	211	5.687203791469194	31.57154563090946	211	5.687203791469194	0.0	31.57154563090946	0.0
base_bert+MLM on our data+QA squad	10.42654028436019	38.48925966485835	211	10.42654028436019	38.48925966485835	211	10.42654028436019	0.0	38.48925966485835	0.0
Base Bert+finetune Q/A on our 200 Q/A	48.34123222748815	71.50455989422866	211	48.34123222748815	71.50455989422866	211	48.34123222748815	0.0	71.50455989422866	0.0
base_bert+MLM on our data+QA squad+our 200 Q/A	50.71090047393365	74.54551157490803	211	50.71090047393365	74.54551157490803	211	50.71090047393365	0.0	74.54551157490803	0.0
Baseline Q/A Squeeze Bert	4.739336492890995	31.57909317121808	211	4.739336492890995	31.57909317121808	211	4.739336492890995	0.0	31.57909317121808	0.0
Squeeze Bert+MLM on our data+QA squad	3.791469194312798	25.86135503016668	211	3.7914691943127	25.86135503016	211	3.79146919431	0.0	25.861355030166404	0.0
Squeeze Bert+finetune Q/A on our 200 Q/A	44.54976303317535	70.72189862957234	211	44.54976303317535	70.72189862957234	211	44.54976303317535	0.0	70.72189862957234	0.0
Squeeze Bert+MLM on our data+QA squad+our 200 Q/A	37.91469194312796	65.00413201337508	211	37.91469194312796	65.00413201337508	211	37.91469194312796	0.0	65.00413201337508	0.0