How can I benchmark my Machine Learning forecasting procedure?

[rosho]

Hi
I am forecasting the number of logins. My dataset has the number of logins by the hour (1 month).

I use a) and b) to clean the data (removing or transforming outliers).
With c) I forecast

(My Machine Learning alert)
a) Standard scaler
b) Detect outliers using DBSCAN
c) Forecast with Kalman filter or MLP

How can I benchmark my results besides from using a test set? Maybe setting an alert (I will call it SPL alert) and later compare it with my Machine Learning alert and check which one failed more?

Do you have any suggestions?

Thank you

[gbenitezfelix]

The easiest way to create a benchmark of your model is to use a naive forecasting. This prediction will always use the last period as predicted. You can read about it here https://otexts.com/fpp2/simple-methods.html.

once you implement it you can calculate the error of each forecasting model. If the naive forecast has lower error than your initial model you should consider to tune that model