Combining Bayesian VARs with survey density forecasts: does it pay off?

Marta Bańbura, Federica Brenna, Joan Paredes, Francesco Ravazzolo

Working Paper Series

Combining Bayesian VARs with survey density forecasts: does it pay off?

No 2543 / May 2021

Disclaimer: This paper should not be reported as representing the views of the European Central Bank (ECB). The views expressed are those of the authors and do not necessarily reflect those of the ECB.

Abstract

This paper studies how to combine real-time forecasts from a broad range of Bayesian vector autoregression (BVAR) speci cations and survey forecasts by optimally exploiting their properties. To do that, it compares the forecasting performance of optimal pooling and tilting techniques, including survey forecasts for predicting euro area ination and GDP growth at medium-term forecast horizons using both univariate and multivariate forecasting metrics. Results show that the Survey of Professional Forecasters (SPF) provides good point forecast performance, but also that SPF forecasts perform poorly in terms of densities for all variables and horizons. Accordingly, when the model combination or the individual models are tilted to SPF's rst moments, point accuracy and calibration improve, whereas they worsen when SPF's second moments are included. We conclude that judgement incorporated in survey forecasts can considerably increase model forecasts accuracy, however, the way and the extent to which it is incorporated matters.

Keywords: Real Time, Optimal Pooling, Judgement, Entropic tilting, Survey of Professional Forecasters

JEL Codes: C11, C32, C53, E27, E37

ECB Working Paper Series No 2543 / May 2021

1

Non-technical summary

Analytical models can be used to obtain point forecasts and the uncertainty surrounding them. The latter is as important (in some instances more so) than the point prediction, since it allows academics and policy-makers to gauge the probability of downside versus upside risks to the economy. In the time series realm, Bayesian Vector Auto-Regression models (BVARs) became a standard tool for forecasting and scenario analysis at central banks. As any other forecasting tool, there is a large variation in performance across di erent speci cations. It is dicult to choose an individual model which, given prior information, set of variables, and data transfor- mations, is able to forecast equally well every variable, at every horizon, and at each point of the business cycle.

For this reason, with the aim of nding the best possible forecasts for our two target variables, the year-on-year growth rates of euro area real Gross Domestic Product (GDP) and HICP ination, in this paper we optimally combines di erent sources of information. First, using real-time data, we estimate a wide set of BVAR speci cations and evaluate their forecasts over the sample 2000-2019. The models di er on data set size and composition, data transformation, degree of time variation, prior speci cation, and inclusion of o -model information. We then combine forecasts from these models using a method called linear optimal pooling. The method nds weights associated with each model by maximising past predictive performances (based on individual variables or on both variables together). Finally, we test whether combining forecasts from the euro area Survey of Professional Forecasters (SPF) with the model forecasts further improves the forecast performance.

We nd that including the complete distribution from SPF in an optimal pool slightly improves nal performance at the one-year horizon; we also nd that including only information on certain moments via entropic tilting from the SPF has di erent e ects depending on the moment included: the rst moment improves overall density forecast accuracy, whereas the second one always worsens it. This is due to the fact that SPF respondents tend to be overcon dent, and provide too narrow forecast densities, which in turn result in poor calibration and low predictive scores. In contrast, the point forecasts from SPF tend to have smaller errors than those from the BVARs. Therefore, including only information on survey means can substantially improve our forecasts both from a point and a density perspective.

ECB Working Paper Series No 2543 / May 2021

2

• Introduction

Optimally combining forecasts from multiple models in order to robustly predict future paths of macroeconomic variables is a methodology, which has been advocated for some time in the economic literature (for a comprehensive review, see Bassetti et al., 2020). The reason for this is that it is hard to nd an individual model, which can be considered the best performing" in all possible forecasting dimensions, i.e. for any variable, at any forecast horizon, at any point in history, and for any loss function metric (be it in terms of point or density forecast). It is then quite natural to think about combinations as a way of averaging multiple measurements of the same outcome. These measurements may be the result of known econometric models, or they may come from a mixture of unobserved data, models, and judgement calls, such as the gures published in survey forecasts.

Having the above in mind, we construct a real-time forecast exercise for the euro area real GDP and HICP ination using econometric models at one- and two-year-ahead forecast hori- zons. We use several types of Bayesian Vector Auto-Regression models (BVARs), which became a standard tool for forecasting and scenario analysis in the central banking community, see

Banbura et al. (2010) and Karlsson (2013), above all for mid- and long-term forecast horizons. We choose several speci cations, which di er on certain modelling choices, such as data set size and composition, data transformation, degree of time variation, prior speci cation, and inclusion of o -model information. In particular, we include BVAR models with Minnesota (Sims and

Zha, 1998; Banbura et al., 2010) and democratic priors (Villani, 2009; Wright, 2013) and a model with time-varying parameters (Primiceri, 2005). We also include a model proposed by Banbura and van Vlodrop (2018) with a local mean and a univariate unobserved component model with stochastic volatility in the style of Stock and Watson (2007). For some models we use both a 3 and a 19 variable speci cation, resulting overall in 13 di erent BVARs. As mentioned above, we then hedge against model uncertainty by combining those model forecasts by means of linear optimal pooling, where weights are selected in order to maximise forecast accuracy (Hall and Mitchell, 2007; Jore et al., 2010; Geweke and Amisano, 2011; Amisano and Geweke, 2017), both for each individual variable and for the aggregate of the two variables.

We evaluate the performance of individual models and their combinations over the period 2000-2019 at the one- and two-year-ahead horizons, in terms of point and density forecast accuracy. We calculate Root Mean Square Forecast Errors (RMSFE), Log Predictive Scores

ECB Working Paper Series No 2543 / May 2021

3

(LPS) and Continuous Ranked Probability Scores (CRPS) as performance measures. In order to assess calibration, we also compute the Probability Integral Transforms (PITs) and perform a test for the uniformity of the PITs distribution (Berkowitz, 2001). The latter feature is often overlooked in forecast evaluations, although it is key when accurate measures of uncertainty around the point forecasts are needed. We nd that combination improves on individual models, however, it does not achieve good calibration for both, variables and horizons. We then turn to an additional source of information, namely the Survey of Professional Forecasters (SPF), whose forecasts are known for having a good point forecast performance (Kenny et al., 2014). We construct a continuous distribution from the SPF histograms in order to assess their accuracy and calibration. We nd, as expected, high accuracy for the SPF point forecasts, but poor performance from a density perspective, both in terms of accuracy and calibration.

Given that from this rst analysis there is no clear best" forecast between a combination of BVARs and survey forecasts, we then investigate two alternative approaches to incorporate information from both methods in a unique forecast density, hence bene ting from more subjective SPF forecasts, containing forward-looking information, and from the more academic and model based BVAR forecasts, which are mostly based on backward looking information.

In the rst approach, after simulating draws from the SPF distributions we obtain an SPF density forecast, which we incorporate into the pool of models used for the optimal combination. The second method is entropic tilting. Namely, we tilt either the individual models before combining them, or the model combination, to either the rst moment or both rst and second moments of the SPF. Therefore, we extend the literature that applies tilting to individual models (Kruger et al., 2017; Altavilla et al., 2017; Ganics and Odendahl, 2021) and model combinations (Galvao et al., 2021) or just combines macroeconomic models (Amisano and Geweke, 2017). To our knowledge it has not yet been considered to combine tilted forecasts or to include survey forecasts in macroeconomic model density combinations.

We nd that incorporating survey information in model combinations improves forecast performance and calibration, especially in the tilting method, albeit only limited to the rst survey moment. When the individual models or the combination are tilted to both mean and variance of the SPF, there is a general worsening of the performance. Our results are similar to Galvao et al. (2021) for U.K.'s GDP and ination; they also found that judgement on the mean tends to improve model density forecasts at short horizons, whereas survey second moments hinder performance at short horizons. Combining individual BVARs both with and without

ECB Working Paper Series No 2543 / May 2021

4