Assignment 4 Part I: Test Set Error is Lower than the Training Set Error

[violetwhitney]

In Part 1 of the Assignment I adjusted the Hyper Parameters to simplify the model, in order to not overfit the model to the training set.

My Hyper Parameters

However, in my case the test set error is lower than the training set error, rather than the other way around.

My Error Rate

In all other examples, the training set error is more accurate (lower) than the test set. Am I doing something wrong?

Other Example of Error Rate

[mengchantang]

I have the same problem on both parts of the assignment. This is quite counterintuitive.

[danilnagy]

Hey guys, these area great questions and valid concerns. The curves I drew in class represent the typical case, where performance over the test set will be somewhat worse than on the training set. This makes sense because, since the algorithm is learning from the training set it is always inherently biased to perform better on that set than on one it's never seen before.

However, this is definitely not a rule, and from time to time you do see cases where you actually get better performance on the test set. This usually doesn't mean anything specific about your model setup, and does not imply poor performance. However, it is more likely to happen with smaller data sets, such as the ones we are using for demonstrations in the class. This is because any given model is more likely to 'guess right' on a small set of samples even if the model is not that good.

In the real world you would not typically use such advanced algorithms for such small sets of data, and are thus less likely to see this issue in real world cases where you might be using several orders of magnitude more data.

Hope this helps!

[violetwhitney]

Yep! Thanks!