Reproducing the results for vehicle_trips_dataset_without_missing_values #14
Description
Hi, thank you for the hard work establishing this repository, collecting the datasets and releasing the code!
I would like to follow up on the discussion that we had in #11 (comment) about reproducing the results reported on https://forecastingdata.org/.
In short, I am unable to reproduce the MASE scores for vehicle_trips_dataset_without_missing_values reported in the table using the R code provided on GitHub.
| TBATS | SES | Theta | ETS | ARIMA | PR | |
|---|---|---|---|---|---|---|
| My experiment | 1.856 | 2.273 | 1.914 | 1.964 | 2.051 | 2.196 |
| Published | 1.860 | 1.224 | 1.244 | 1.305 | 1.282 | 1.212 |
For mean/median sMAPE/MAE (and all other non-seasonal metrics) the results for PR and SES are identical to the paper, but for ETS and Theta the numbers are quite different from the paper.
I did some more investigation with git blame on the file experiments/feature_functions.R, and it seems that the problem might be caused by some past changes that incorrectly set SEASONALITY_VALS used when computing the results for the vehicle_trips dataset.
If I change SEASONALITY_VALS[[7]] <- 7 to SEASONALITY_VALS[[7]] <- 24, then both MASE results for all models & sMAPE/MAE scores for Theta and ETS get very close to the ones published in the appendix.
Can you please tell me if there are mistakes in my reasoning and, if so, what I should change in my setup to obtain the correct results comparable to the official table?