Accuracy measures for a cross-validation model and a conformal prediction model

Return range of summary measures of the out-of-sample forecast accuracy. If x is given, the function also measures test set forecast accuracy. If x is not given, the function only produces accuracy measures on validation set.

Usage

# Default S3 method
accuracy(
  object,
  x,
  CV = TRUE,
  period = NULL,
  measures = interval_measures,
  byhorizon = FALSE,
  ...
)

Arguments

object: An object of class "cvforecast" or "cpforecast".
x: An optional numerical vector containing actual values of the same length as mean in object.
CV: If TRUE, the cross-validation forecast accuracy will be returned.
period: The seasonal period of the data.
measures: A list of accuracy measure functions to compute (such as point_measures or interval_measures).
byhorizon: If TRUE, accuracy measures will be calculated for each individual forecast horizon h separately.
...: Additional arguments depending on the specific measure.

Value

A matrix giving mean out-of-sample forecast accuracy measures.

Details

The measures calculated are:

ME: Mean Error
MAE: Mean Absolute Error
MSE: Mean Squared Error
RMSE: Root Mean Squared Error
MPE: Mean Percentage Error
MAPE: Mean Absolute Percentage Error
MASE: Mean Absolute Scaled Error
RMSSE: Root Mean Squared Scaled Error
winkler_score: Winkler Score
MSIS: Mean Scaled Interval Score

Examples

# Simulate time series from an AR(2) model
library(forecast)
series <- arima.sim(n = 1000, list(ar = c(0.8, -0.5)), sd = sqrt(1))

# Cross-validation forecasting with a rolling window of length 100
far2 <- function(x, h, level) {
  Arima(x, order = c(2, 0, 0)) |>
    forecast(h = h, level)
}
fc <- cvforecast(series, forecastfun = far2, h = 3, level = c(80, 95),
                 forward = TRUE, initial = 1, window = 100)

# Out-of-sample forecast accuracy on validation set
accuracy(fc, measures = point_measures, byhorizon = TRUE)
#>                  ME       MAE      MSE      RMSE        MPE     MAPE      MASE
#> CV h=1 -0.007612522 0.8151254 1.017957 0.8151254  0.7657986 428.6975 0.7580282
#> CV h=2 -0.011786774 1.0713234 1.746509 1.0713234 16.6966949 308.4014 0.9958467
#> CV h=3 -0.008274060 1.0712870 1.776563 1.0712870 15.9913942 298.7024 0.9958713
#>            RMSSE
#> CV h=1 0.6077954
#> CV h=2 0.7987764
#> CV h=3 0.7987887
accuracy(fc, measures = interval_measures, level = 95, byhorizon = TRUE)
#>        Winkler_95  MSIS_95
#> CV h=1   4.676458 4.347321
#> CV h=2   6.170746 5.736393
#> CV h=3   6.328972 5.882909

# Out-of-sample forecast accuracy on test set
accuracy(fc, x = c(1, 0.5, 0), measures = interval_measures,
         level = 95, byhorizon = TRUE)
#>          Winkler_95  MSIS_95
#> CV h=1     4.676458 4.347321
#> CV h=2     6.170746 5.736393
#> CV h=3     6.328972 5.882909
#> Test h=1   3.892988 3.596556
#> Test h=2   5.263013 4.862260
#> Test h=3   5.330563 4.924667