diff --git a/bmnr/index.html b/bmnr/index.html index 1d0036d3..d3a69b69 100644 --- a/bmnr/index.html +++ b/bmnr/index.html @@ -1,9 +1,11 @@ -The Big Macro Nowcasting Ranking | Haris Karagiannakis

The Big Macro Nowcasting Ranking

List of forecasting models

AcronymModel Description
AR(P)Autoregressive iterated-specification
RWRandom walk
ARDI(K,BIC)Autoregressive diffusion indices with K factors. Optimal lag-order via BIC
T.ARDI(K,BIC)ARDI with target-factors. Hard-threshold set to |t-stat|>1.96
BVAR-MinnHomoscedastic large Bayesian VAR
BVAR-CSVLarge Bayesian VAR with heteroscedastic innovations
BBoostQuadratic-loss L2 boosting, block-wise
CBoostQuadratic-loss L2 boosting, component-wise
CSRComplete Subset Regressions (20C4) with hard-thresholding preselection
BagLinear bagging with hard-thresholding preselection
BTreeBoosting regression trees
RFRandom forests
SVRSupport vector machine regression with Gaussian Kernel function
RidgeRidge regression
LASSOLeast absolute shrinkage and selection operator with BIC for lambda
AdaLASSOAdaptive LASSO
ENElastic Net
AdaENAdaptive EN
LSTMLong-short-term memory RNN with 3-hidden layers
SgLASSOSparse-group-LASSO-MIDAS with block-K-fold CV for lambda & alpha

List of methods for treating mixed-frequencies

AcronymTransformation Description
D1Down-sampling to Quarterly (Temporal aggregation with equal-weights)
D2Unrestricted MIDAS Polynomials
D3Legendre Polynomials (3rd degree)

The Ranking: Real-time model evaluation

Modelsn=0n=1n=2n=3n=4avg
Bag-D30.7280.9320.9720.9980.9940.925
CSR-D20.7070.9380.97611.0040.925
BBoost-D1F0.7250.9250.9781.0061.0010.927
Bag-D2F0.7720.910.9670.9940.9990.929
Ridge-D10.7640.9240.9580.9961.0010.929
Ridge-D20.7420.9260.9721.0011.0050.929
Bag-D20.7670.9210.9690.9931.0020.93
T.ARDI(2)0.7450.9290.9751.0021.0070.932
Bag-D1F0.7590.9250.971.0051.0010.932
Bag-D10.7830.9180.9610.9961.0020.932
CSR-D10.7580.9340.9741.0041.0090.936
Ridge-D30.7910.9210.9691.0041.0030.938
RF-D1F0.8160.9170.9660.9950.9920.938
Ridge-D2F0.8080.9130.9680.9991.0040.938
CBoost-D1F0.7730.9250.9831.011.0010.938
SVR-D10.8250.9190.9640.9920.9950.939
LASSO-D1F0.7870.930.9781.0020.9990.939
AdaLASSO-D1F0.7810.9370.981.0030.9980.94
RF-D10.8040.9240.9710.9991.0030.94
Bag-D3F0.8160.9170.9721.0050.9930.941
EN-D2F0.8160.9210.9690.9991.0010.941
EN-D1F0.7990.9280.9781.00110.941
SVR-D30.8270.920.970.9950.9970.942
RF-D30.8120.9360.9690.9941.0010.942
ARDI(1)0.7990.9390.9750.9991.0010.943
CBoost-D2F0.8210.920.9650.9991.0080.943
BBoost-D2F0.8360.9120.9670.99810.943
SVR-D3F0.8310.9140.9720.9951.0010.943
SVR-D20.8280.9230.9710.9970.9970.943
AdaEN-D20.7660.9470.9771.0031.0240.943
EN-D20.7770.9710.9780.9960.9940.943
RF-D20.8160.9250.9761.00110.943
LASSO-D2F0.8310.920.9690.99810.944
AdaEN-D1F0.8020.9340.9811.00210.944
SVR-D1F0.8270.9210.9710.99910.944
CSR-D2F0.8240.9280.970.99710.944
SVR-D2F0.8310.9250.9720.9960.9950.944
RF-D2F0.8270.9290.9720.9980.9950.944
BTree-D1F0.8110.9230.9751.0021.0080.944
AdaEN-D2F0.8340.9260.9650.9991.0020.945
AdaLASSO-D2F0.840.9220.9660.9981.0010.945
AdaLASSO-D20.7650.9540.9790.9981.0340.946
LSTM-D1F0.8450.9180.9681.0050.9940.946
ARDI(2)0.7980.9470.9781.0011.0080.946
EN-D30.8340.9240.975110.947
CSR-D30.8140.9310.981.0051.0030.947
RW0.8390.9210.9750.9990.9990.947
LSTM-D2F0.8440.9280.96610.9970.947
LASSO-D20.780.980.98310.9970.948
BVAR-CSV0.8110.9260.981.0121.0130.948
RF-D3F0.8440.9340.9760.9911.0010.949
T.ARDI(1)0.8140.9480.9751.0051.0070.95
AdaEN-D10.8170.9310.9911.0071.0030.95
BBoost-D3F0.8650.9060.9731.0061.0010.95
AdaEN-D30.8480.9280.97611.0020.951
LSTM-D3F0.8510.9260.9720.9941.0130.951
SgLASSO-D30.8490.9250.9781.0031.0050.952
SgLASSO-D3F0.8610.9120.9771.0111.0030.953
CSR-D1F0.8770.9250.970.9980.9990.954
EN-D10.8090.9360.9441.0451.0360.954
BTree-D10.8120.9330.9881.0151.0240.955
AdaEN-D3F0.8830.9240.9681.0010.9990.955
LSTM-D30.8580.9320.9781.0051.0040.955
BVAR-Minn0.850.9330.9741.0121.0150.957
EN-D3F0.8910.9260.97110.9990.957
CBoost-D3F0.8540.9360.9851.0131.0010.958
LSTM-D10.8660.9360.9781.0021.0130.959
CSR-D3F0.890.9380.971.0010.9970.959
LASSO-D3F0.9030.9210.97111.0030.96
AdaLASSO-D3F0.8930.9290.9711.0031.0040.96
BTree-D2F0.850.9540.9911.0170.9970.962
LSTM-D20.8680.9430.9861.0170.9990.962
BTree-D20.8390.9520.9941.0121.0160.963
Ridge-D3F0.8020.9411.0171.0461.0330.968
AdaLASSO-D10.830.9471.0191.0371.0220.971
BTree-D30.8510.9880.991.0161.0210.973
Ridge-D1F0.7790.9591.0261.061.0540.976
LASSO-D30.8320.9321.0181.061.0560.98
AdaLASSO-D30.8340.9451.031.0651.060.987
LASSO-D10.7741.0070.9871.1021.0740.989
BTree-D3F0.881.0310.9991.011.0330.991
AR(BIC)1.0081.00311.00111.002
AR(CV)1.021.00411.0010.9991.005
AR(4)1.0491.0171.0020.9990.9991.013
AR(1)NaNNaNNaNNaNNaNNaN

Forecasting Performance by TransformationAutomatic

The Big Macro Nowcasting Ranking

The objective of this page is to maintain and populate the ranking below, with newly introduced statistical models in an effort to provide a continuously updated comprehensive comparison of methodologies for nowcasting and forecasting macroeconomic activity. The model comparison is based on a set of (pseudo) real-time vintages, formed using a rich standardized set of variables at mixed-frequencies.

The performance evaluation currently covers US real GDP growth rate (QoQ%), comparing 20 distinct methodologies (including ML, as well as standard econometric techniques and workhorse benchmarks), which are combined with three data transformations (D1,D2,D3) for taking into account the mixed-frequency dimension. This results into a ranking containing a total of 85 specifications.

The dataset consists of 87 quarterly, and 171 monthly (including 31 financial market) indicators. The indicators correspond to the series found in the FRED-MD and -QD datasets and were downloaded at their original (i.e. highest-available) sampling frequency, whenever possible. Publication release delays were inferred from the metadata in the FRED database (https://fred.stlouisfed.org) and then applied to each series to mimic the ragged-edge structure the forecaster would face in reality. The out-of-sample evaluation uses end-of-month vintages at a monthly periodicity, assuming that economic activity is monitored in \textit{real-time} by updating the projections at the end of every month.

A list of the indicators that compose the mixed-frequency dataset, can be accessed +here. +SeriesID refers to the FRED mnemonic, while the release delay (RDelay) measures the approximate number of days it takes for the respective indicator to be released after the closing of the reference month or quarter. The monthly mixed-frequency vintages used in the POOS model evaluation can be downloaded here. The full set consists of 376 end-of-month vintages spanning the period 19900131-20210430, including monthly and quarterly unbalanced panels whose ragged edge has been imposed by applying the inferred publication delays.

List of forecasting models

AcronymModel Description
AR(P)Autoregressive iterated-specification
RWRandom walk
ARDI(K,BIC)Autoregressive diffusion indices with K factors. Optimal lag-order via BIC
T.ARDI(K,BIC)ARDI with target-factors. Hard-threshold set to |t-stat|>1.96
BVAR-MinnHomoscedastic large Bayesian VAR
BVAR-CSVLarge Bayesian VAR with heteroscedastic innovations
BBoostQuadratic-loss L2 boosting, block-wise
CBoostQuadratic-loss L2 boosting, component-wise
CSRComplete Subset Regressions (20C4) with hard-thresholding preselection
BagLinear bagging with hard-thresholding preselection
BTreeBoosting regression trees
RFRandom forests
SVRSupport vector machine regression with Gaussian Kernel function
RidgeRidge regression
LASSOLeast absolute shrinkage and selection operator with BIC for lambda
AdaLASSOAdaptive LASSO
ENElastic Net
AdaENAdaptive EN
LSTMLong-short-term memory RNN with 3-hidden layers
SgLASSOSparse-group-LASSO-MIDAS with block-K-fold CV for lambda & alpha

List of methods for treating mixed-frequencies

AcronymTransformation Description
D1Down-sampling to Quarterly (Temporal aggregation with equal-weights)
D2Unrestricted MIDAS Polynomials
D3Legendre Polynomials (3rd degree)

The Ranking: Real-time model evaluation

The table reports RMSE’s relative to the AR(1) benchmark for the n-quarters ahead prediction, with n=0 reflecting the nowcast. The models have been ranked wrt the last column which corresponds to the average relative RMSE over all 5 horizons. The real-time POOS evaluation is based on 220 monthly vintages over the period Jan-2003 to Apr-2021. D1 denotes single-frequency information set; D2 U-MIDAS polynomials; and D3 Legendre polynomials. The table ranks specifications coming from the different combinations of ML models with all the transformations D1, D2 and D3 plus their factor-only counterparts. Models with an acronym ending in ‘F’ contain only factors on the RHS.

Modelsn=0n=1n=2n=3n=4avg
Bag-D30.7280.9320.9720.9980.9940.925
CSR-D20.7070.9380.97611.0040.925
BBoost-D1F0.7250.9250.9781.0061.0010.927
Bag-D2F0.7720.910.9670.9940.9990.929
Ridge-D10.7640.9240.9580.9961.0010.929
Ridge-D20.7420.9260.9721.0011.0050.929
Bag-D20.7670.9210.9690.9931.0020.93
T.ARDI(2)0.7450.9290.9751.0021.0070.932
Bag-D1F0.7590.9250.971.0051.0010.932
Bag-D10.7830.9180.9610.9961.0020.932
CSR-D10.7580.9340.9741.0041.0090.936
Ridge-D30.7910.9210.9691.0041.0030.938
RF-D1F0.8160.9170.9660.9950.9920.938
Ridge-D2F0.8080.9130.9680.9991.0040.938
CBoost-D1F0.7730.9250.9831.011.0010.938
SVR-D10.8250.9190.9640.9920.9950.939
LASSO-D1F0.7870.930.9781.0020.9990.939
AdaLASSO-D1F0.7810.9370.981.0030.9980.94
RF-D10.8040.9240.9710.9991.0030.94
Bag-D3F0.8160.9170.9721.0050.9930.941
EN-D2F0.8160.9210.9690.9991.0010.941
EN-D1F0.7990.9280.9781.00110.941
SVR-D30.8270.920.970.9950.9970.942
RF-D30.8120.9360.9690.9941.0010.942
ARDI(1)0.7990.9390.9750.9991.0010.943
CBoost-D2F0.8210.920.9650.9991.0080.943
BBoost-D2F0.8360.9120.9670.99810.943
SVR-D3F0.8310.9140.9720.9951.0010.943
SVR-D20.8280.9230.9710.9970.9970.943
AdaEN-D20.7660.9470.9771.0031.0240.943
EN-D20.7770.9710.9780.9960.9940.943
RF-D20.8160.9250.9761.00110.943
LASSO-D2F0.8310.920.9690.99810.944
AdaEN-D1F0.8020.9340.9811.00210.944
SVR-D1F0.8270.9210.9710.99910.944
CSR-D2F0.8240.9280.970.99710.944
SVR-D2F0.8310.9250.9720.9960.9950.944
RF-D2F0.8270.9290.9720.9980.9950.944
BTree-D1F0.8110.9230.9751.0021.0080.944
AdaEN-D2F0.8340.9260.9650.9991.0020.945
AdaLASSO-D2F0.840.9220.9660.9981.0010.945
AdaLASSO-D20.7650.9540.9790.9981.0340.946
LSTM-D1F0.8450.9180.9681.0050.9940.946
ARDI(2)0.7980.9470.9781.0011.0080.946
EN-D30.8340.9240.975110.947
CSR-D30.8140.9310.981.0051.0030.947
RW0.8390.9210.9750.9990.9990.947
LSTM-D2F0.8440.9280.96610.9970.947
LASSO-D20.780.980.98310.9970.948
BVAR-CSV0.8110.9260.981.0121.0130.948
RF-D3F0.8440.9340.9760.9911.0010.949
T.ARDI(1)0.8140.9480.9751.0051.0070.95
AdaEN-D10.8170.9310.9911.0071.0030.95
BBoost-D3F0.8650.9060.9731.0061.0010.95
AdaEN-D30.8480.9280.97611.0020.951
LSTM-D3F0.8510.9260.9720.9941.0130.951
SgLASSO-D30.8490.9250.9781.0031.0050.952
SgLASSO-D3F0.8610.9120.9771.0111.0030.953
CSR-D1F0.8770.9250.970.9980.9990.954
EN-D10.8090.9360.9441.0451.0360.954
BTree-D10.8120.9330.9881.0151.0240.955
AdaEN-D3F0.8830.9240.9681.0010.9990.955
LSTM-D30.8580.9320.9781.0051.0040.955
BVAR-Minn0.850.9330.9741.0121.0150.957
EN-D3F0.8910.9260.97110.9990.957
CBoost-D3F0.8540.9360.9851.0131.0010.958
LSTM-D10.8660.9360.9781.0021.0130.959
CSR-D3F0.890.9380.971.0010.9970.959
LASSO-D3F0.9030.9210.97111.0030.96
AdaLASSO-D3F0.8930.9290.9711.0031.0040.96
BTree-D2F0.850.9540.9911.0170.9970.962
LSTM-D20.8680.9430.9861.0170.9990.962
BTree-D20.8390.9520.9941.0121.0160.963
Ridge-D3F0.8020.9411.0171.0461.0330.968
AdaLASSO-D10.830.9471.0191.0371.0220.971
BTree-D30.8510.9880.991.0161.0210.973
Ridge-D1F0.7790.9591.0261.061.0540.976
LASSO-D30.8320.9321.0181.061.0560.98
AdaLASSO-D30.8340.9451.031.0651.060.987
LASSO-D10.7741.0070.9871.1021.0740.989
BTree-D3F0.881.0310.9991.011.0330.991
AR(BIC)1.0081.00311.00111.002
AR(CV)1.021.00411.0010.9991.005
AR(4)1.0491.0171.0020.9990.9991.013
AR(1)NaNNaNNaNNaNNaNNaN

The graph provides a visualization of the horse race. Furthermore, it adds a second crucial metric for measuring comparative performance, the MAE. The axes show the relative error measures (RMSE and MAE) averaged over all 5 horizons. Candidate models that are closest to the origin (-bottom left) are the best performers.

Forecasting Performance by Transformation

You might also find the Plotly JSON Editor useful.

Math

Wowchemy supports a Markdown extension for $\LaTeX$ math. You can enable this feature by toggling the math option in your config/_default/params.yaml file.

To render inline or block math, wrap your LaTeX math with {{< math >}}$...${{< /math >}} or {{< math >}}$$...$${{< /math >}}, respectively. (We wrap the LaTeX math in the Wowchemy math shortcode to prevent Hugo rendering our math as Markdown. The math shortcode is new in v5.5-dev.)

Example math block:

{{< math >}}
+    - Math: $x = {-b \pm \sqrt{b^2-4ac} \over 2a}$

Charts

Wowchemy supports the popular Plotly format for interactive charts.

Save your Plotly JSON in your page folder, for example line-chart.json, and then add the {{< chart data="line-chart" >}} shortcode where you would like the chart to appear.

Demo:

You might also find the Plotly JSON Editor useful.

Math

Wowchemy supports a Markdown extension for $\LaTeX$ math. You can enable this feature by toggling the math option in your config/_default/params.yaml file.

To render inline or block math, wrap your LaTeX math with {{< math >}}$...${{< /math >}} or {{< math >}}$$...$${{< /math >}}, respectively. (We wrap the LaTeX math in the Wowchemy math shortcode to prevent Hugo rendering our math as Markdown. The math shortcode is new in v5.5-dev.)

Example math block:

{{< math >}}
 $$
 \gamma_{n} = \frac{ \left | \left (\mathbf x_{n} - \mathbf x_{n-1} \right )^T \left [\nabla F (\mathbf x_{n}) - \nabla F (\mathbf x_{n-1}) \right ] \right |}{\left \|\nabla F(\mathbf{x}_{n}) - \nabla F(\mathbf{x}_{n-1}) \right \|^2}
 $$