Omniglot: the task characters should not come from the same alphabet.

[cjauvin]

First of all thank you for your nice notebooks, I find them quite useful!

I think one problem with your Omniglot notebook is that the characters for a given task all come from the same alphabet. I'm not 100% sure, but I see two reasons why it should be otherwise:

1. In Section 3.1 of the ProtoNet paper, it says:

We trained prototypical networks using Euclidean distance in the 1-shot and 5-shot scenarios with
training episodes containing 60 classes and 5 query points per class.

Since 60 is greater than the average alphabet size, it makes me think that the characters are coming from many alphabets.

2. In Section 4.1.1 of the Matching Networks paper, it says:

The N-way Omniglot task setup is as follows: pick N unseen character classes, independent of alphabet, as L.

Perhaps that could explain why your results do not match those of the ProtoNet paper?

[frenchie4111]

@cjauvin thanks for the comment, it's cool to see other people finding value from my work.

I noticed that discrepancy as well, it seems to me like the original Omniglot paper used only characters from the same language in the test set (see Fig 5), while the ProtoNet and Matching Networks use characters from any language.

I chose to use the test method from the original Omniglot paper because it seems like a harder and more applicable problem. It makes sense to me that the ProtoNet is performing worse on my test set, as the problem is much harder with characters from the same language, than with characters from any language.