Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is a classification model. The basic idea of LDA is that it assumes, for each class $k$, that $f_k(X)$ is normal (in the case of $p$ = 1) or multivariate normal (if $p>1$).

That is,

$$X\sim \mathcal{N}({\mu }_k,\Sigma )$$

In LDA, the mean vectors ${\mu }_k$ differ for each class, but we assume the same covariance matrix $\Sigma$ across all classes.

To estimate ${\mu }_k$, we simply take the sample means of each predictor in each class $k$.

To estimate $\Sigma$, we take the weighted average of the sample covariance matrices for each of the $k$ classes.

Once we have estimate of these parameters, we assign an observation $X=x$ to the most likely class, i.e. the class for which

$${\delta }_k(x)=x^T{\Sigma }^{-1}{\mu }_k-\frac{1}{2}{\mu }_k^T{\Sigma }^{-1}{\mu }_k+\log {\pi }_k$$

is the largest, where ${\pi }_k$ is the prior probability (base rate) for class $k$.