Adding a baseline model(s)

[IsaacMacarthur]

One suggestion that has come up in discussions on how to improve the hub is the idea of adding a baseline model(s) for comparison. The scoring metric that the hub is currently using, the energy score, can be more easily interpreted if there is a baseline model with baseline scores that other models can be compared against.

We had one suggestion for a possible baseline model that would be fit using the following steps:

1. We find the national proportion averages by variant over the last n weeks, for some n.
2. Find the changes in the national proportion over the last m weeks by variant, and calculate the empirical distribution of the differences.
3. For every state, to generate one forecasted sample H days into the future for one variant, take the national proportion average and add H draws from the empirical distribution of the differences. We can then softmax over the variants to ensure they sum to 1.

We would appreciate any feedback, both on the baseline model suggested here and on other possible baseline models that would be useful.

[elray1]

Here are two variations another idea with a different flavor:

1. Fit a multinomial logistic regression to the total aggregated reported counts at the national level over the past $n$ days using maximum likelihood (i.e., frequentist methods to get point estimates). Predictions will be the same for all states.
   1. Variation 1 is a standard MLR model in time: $c_t | \pi_t, n_t \sim Multinomial(\pi_t, n_t)$ where $\pi_{t} = \text{softmax}(\gamma_t)$ and $\gamma_{t,v} = \beta_{0,v} + \beta_{1,v} t$, with separate coefficients for each variant $v$.
   2. Variation 2 throws away the slope: $c_t | \pi_t, n_t \sim Multinomial(\pi_t, n_t)$ where $\pi_{t} = \text{softmax}(\gamma_t)$ and $\gamma_{t,v} = \beta_{0,v}$, with separate intercepts for each variant $v$
2. Point predictions are obtained by getting predictions from this model
3. Probabilistic sample predictions are obtained by somehow bootstrapping prediction errors that the models under point 1 would have had in past weeks. Haven't thought through how to deal with the multinomial sampling bit.

[dylanhmorris]

Recording my suggestion from our f2f discussion that the error distribution be chosen to optimize retrospective coverage.