02.MA, AR and ARMA processes.md

MA, AR and ARMA processes

We will see these families of $SSP$:

• MA (Moving Average): Use past errors to predict future values.
• AR (Autoregressive): Use past values to predict future values.
• ARMA (Autoregressive Moving Average): Combine both AR and MA elements

Moving averages

With $e(t)\sim wn(0,\lambda^2)$ we can define the moving average process of order $n$ $MA(n)$ in this way:

$$y(t)=c_0\cdot e(t)+c_1e(t-1)+c_2\cdot e(t-2)+\cdots+c_ne(t-n)$$

$MA(n)$ is still stationary . A generalization of $MA(n)$ is:

$$MA(\infty)=\sum_{i=0}^{+\infty}c_ie(t-i), \space e(t)\sim\mathrm{wn}(0,\lambda^{2})$$

but the series must converge! The properties to remember here are:

$$m_y(t)=m_y(0)=0$$

$$\gamma_{y}(\tau)=\begin{cases}\left({c_0}^2+{c_1}^2+{c_2}^2+...+{c_n}^2\right)\cdot\lambda^2&\text{if }\tau=0\\left({c_0}c_1+{c_1}c_2+...+{c{n-1}}c_n\right)\cdot{\lambda^2}&\text{if }\tau=\pm1\\left({c_0}c_2+{c_1}c_3+...+{c_{n-2}}c_n\right)\cdot{\lambda^2}&\text{if }\tau=\pm2\\vdots\\left({c_0}c_n\right)\cdot{\lambda^2}&\text{if }\tau= n\0&\text{if }|\tau|> n\end{cases}$$

So for example, for $y(t)=e(t)+c\cdot e(t-1)$:

$$\gamma_y(0) = 1+ c^2$$ and $$\gamma_y(1) = \gamma_y(-1) = c$$

AR

Autoagressive models will allow us to have $\gamma(\tau) \ne 0$ for all $\tau$ using a finite set of coefficients.

$$y(t)=a_1y(t-1)+a_{2}y(t-2)+\cdots+a_{m}y(t-m)+e(t)$$

with $e(t) \sim wn(0,\lambda ^2)$ . Where $n$ is the order of the process and $e(t)\sim WN(0,\lambda^2)$. Covariance function with $n=1$ (case of $AR(1)){:}$

$$\begin{cases}\frac{1}{1-a^2}\lambda^2&\tau=0\\ \gamma(\tau)=a\gamma(\tau-1)&|\tau|>0\end{cases}$$ well defined if $|a|<1$.

If $|a| > 1$, the system generates a process with variance $\gamma(0) \rightarrow \infty$ .

The $MA(\infty)$ representation of an AR process refers to representing an autoregressive process as an infinite moving average MA process. performing the recursion.

The intuition behind this representation is that an AR process relies on its past values which, in turn, also depend on their own past values, and so on. If you keep substituting back these dependencies, you end up with a representation that expresses the current value of the process as the sum of the effects of all past and current shock terms $\epsilon_t$, each weighted by a $\psi_i$ coefficient.

$$ \begin{array}{rl} y(t) &= \alpha q(t-1) + e(t) \\ y(t) &= q(t-1) + e(t-1) + e(t-2) \\ &= e(t) + ae(t-1) + a^2q(t-2) \\ y(t) &= e(t) + ae(t-1) + a^2[\alpha(t-3) + e(t-2)] \\ &= e(t) + ae(t-1) + a^2e(t-2) + a^3q(t-3) \\ \vdots & \vdots \\ y(t) &= e(t) + ae(t-1) + a^2e(t-2) + ... = \sum_{i=0}^{\infty} a^{i} e(t-i) \quad\leftarrow\text{MA}(\infty) \end{array} $$

$MA(\infty)$ representation

The $MA(\infty)$ representation is mainly of theoretical interest to write the steady-state solution of an AR process but in practical time series modeling is useless.

It's interesting to notice that the constraint $|a| < 1$ is need to check to bound the variance value, which must be finite.

$$\sum_{i=0}^{+\infty}c_i^2=\sum_{i=0}^{+\infty}a^{2i}$$

The variance must be convergent if $|a|<1.$ This is a critical condition for the stability and stationarity of the $AR(1)$ process.

Yule Walker equations

To easily compute the covariance function we can use this "trick" which is known as "Yule Walker equations." For example for $v(t) \sim AR(2)$ : $$v(t+1)=a_{1}v(t-1)+a_{2}v(t-2)+n(t)$$ we can multiply both sides by $v(t-\tau)$ :

$$v(t+1)v(t-\tau)=a_{1}v(t-1)v(t-\tau)+a_{2}v(t-2)v(t-\tau)+n(t)v(t-\tau)$$

Why ? so that we can express it as:

$$\gamma (\tau) = a_{1}\gamma (\tau-1) + a_{2}\gamma (\tau-2) + \begin{cases} 0 & \tau > 0 \\ \lambda^2 & \tau = 0 \end{cases}$$

In this way we can make a system of equations (the Yule Walker equations and compute the covariance function of AR of any order).

ARMA

An ARMA process is a combination of Autoregressive (AR) and Moving Average (MA) models.

$$v(t) = a_1v(t-1)+...+a_{n_a}v(t-n_a) + c_0\eta(t) + ... + c_1\eta(t-1) + ... + c_{n_c}\eta{(t-n_c)}$$

$ARMA(n_a,n_c)$ $processes include all other $AR$ and $MA$ models as special cases. Also we can say that given an $ARMA$ process $(n_a,n_b)$ the covariance function:

• if $n_a > n_b$: $\gamma_y(\tau)$ becomes recursive for $\tau=n_a$
• if $n_a \le n_b$: $\gamma_y(\tau)$ becomes recursive for $\tau=n_b +1$

ARMAX

ARMAX models extends ARMA including an exogenous inputs. ARMAX($m,n,p,k$) with $k$ pure input/output (I/O) delay. $p$ order of the exogenous part:

$$\begin{aligned}y(t)&=a_1y(t-1)+a_2y(t-2)+\cdots+a_my(t-m)+\&+c_0t(t)+c_1e(t-1)+\cdots\cdots c_n e(t-n)+\&+b_0u(t-k)+b_{1}u(t-k-1)+\cdots b_{p}u(t-k-p)\end{aligned}$$

The concept of $e(t)$ is a mathematical artifact representing the stochastic nature of measurements. In reality, $e(t)$ does not exist, but when is the output of a system, both deterministic and stochastic effects need to be considered. The deterministic effect is caused by an input $u$, while the stochastic effect is due to the inherent errors in measurements.

Processes with non-null term

TLDR: the key takeaway is that incorrect results are obtained when dealing with non-zero mean processes if one fails to recognize their non-zero mean status or doesn't read the exercise carefully.

Generally computations with non-zero mean signals are more complex compared to the zero-mean case. However, a tip to make the computation more efficient is to introduce unbiased versions of the processes $\tilde{y}$ and $\tilde{e}$, which are obtained by removing the means directly. Unbiased versions are still useful because:

$$\gamma_{\tilde{y}}(\tau)=\gamma_{\gamma}(\tau)\quad\forall\tau$$

$$\begin{cases}\tilde{y}(t)=y(t)-m_y\\tilde{e}(t)=e(t)-m_e&\end{cases}$$

By writing the equation for $\tilde{y}$, the terms can be simplified to make the computations more manageable.