insideout Get Pro

11 · Forecasting honestly

Pro

Time Series, lesson 11

Forecasting honestly

Out-of-sample tests, walk-forward and the baseline to beat.

You know AR models from "Autoregressive models" and the naive forecast from "Random walk or not".

Part 1. Beat the naive guess

Below is a simulated spread with real but modest memory. An AR model is fitted on the last 100 days, refitted every 5 days, and compared with the naive guess, tomorrow = today. Judge it two ways.

Inside the window, and after itLive

Inside the window it was fitted on, every extra lag looked like an improvement, because each new coefficient bends the fit toward that window's noise. On days it hadn't seen, each extra lag made it worse, and past six lags it lost to the naive guess. The best was a single lag, ahead by about 4%.

In-sample, out-of-sample and walk-forward

Data used to fit a model is in-sample. Data the model never saw is out-of-sample. Walk-forward testing refits on a moving window of the past and forecasts the next stretch, again and again, so every forecast is out of sample, as it would have been in real time.

Your turn

Why not shuffle the days and use ordinary k-fold cross-validation on a time series?

A model trained on Wednesday and Friday, then tested on Thursday, has seen the future, and with autocorrelation the neighbours give Thursday away. Walk-forward keeps time in order.

Part 2. The baseline and the score

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. The baseline and the score
  2. A worked problem
  3. The same thing, for real
See Pro plansI have Pro: sign in
The last lesson in Time SeriesBack to the overviewData that arrives in order, and what it remembers Back to Time Series

Time Series overviewSpotted a mistake? Tell us.