View the
latest recording of this lecture
Linear regression models can be conveniently expressed using matrix
notation.
In this lecture, we will see how results for linear models are much
more easily derived and understood using matrix notation than without
it.
Also note that the matrix approach is what is being done in the
background by all good statistical software including R.
Prediction and The Hat Matrix
The vector of fitted values is given by \(\hat{\boldsymbol{\mu}} = X
\hat{\boldsymbol{\beta}} = X (X^TX)^{-1} X^T \boldsymbol{y}\) and
the vector of residuals by \(\boldsymbol{e} =
\boldsymbol{y} - \hat{\boldsymbol{\mu}}\).
The equation for the fitted values just given, can be re-written
\(\hat{\boldsymbol{\mu}} = H \boldsymbol{y} =
\hat{\boldsymbol{y}}\) where H = X
(XTX)-1XT is often called the
hat matrix because it “puts hats on things”!
We have the equality \(\hat{\boldsymbol{y}}
= \hat{\boldsymbol{\mu}}\) because we use the same values for
prediction and mean estimation.
Covariance Matrices
The variance-covariance (or simple covariance, or dispersion
matrix) has variances down the diagonal and covariances off the
diagonal.
It can be shown that for a matrix M and random vector
\(\boldsymbol{z}\) (of appropriate
dimensions) \[\mbox{Var} (M\boldsymbol{z}) =
M \mbox{Var} (\boldsymbol{z}) M^T.\]
The Covariance Matrix for \(\hat{\boldsymbol{\beta}}\)
The covariance matrix of \(\hat{\boldsymbol{\beta}}\) is
\[\begin{aligned}
\mbox{Var}(\hat{\boldsymbol{\beta}}) = \mbox{Var}[ (X^TX)^{-1} X^T
\boldsymbol{y} ] &=& (X^TX)^{-1} X^T \mbox{Var}(\boldsymbol{y})
[(X^TX)^{-1} X^T]^T\\
&=& (X^TX)^{-1} X^T \sigma^2 I [(X^TX)^{-1} X^T]^T\\
&=& \sigma^2 (X^TX)^{-1} X^TX (X^T X)^{-1}\\
&=& \sigma^2 (X^TX)^{-1}\end{aligned}\]
We can use \(\mbox{Var}(\boldsymbol{y}) =
\sigma^2 I\) since the responses are independent and hence
uncorrelated.
The leading diagonal of this matrix is the basis for the standard
errors of our parameter estimates seen in our regression output. is
LS0tDQp0aXRsZTogIkxlY3R1cmUgMTM6IE1hdHJpY2VzIGFuZCBMaW5lYXIgUmVncmVzc2lvbiBNb2RlbHMiDQpzdWJ0aXRsZTogMTYxLjI1MSBSZWdyZXNzaW9uIE1vZGVsbGluZw0KYXV0aG9yOiAiUHJlc2VudGVkIGJ5IEpvbmF0aGFuIEdvZGZyZXkgPGEuai5nb2RmcmV5QG1hc3NleS5hYy5uej4iICANCmRhdGU6ICJXZWVrIDUgb2YgU2VtZXN0ZXIgMiwgYHIgbHVicmlkYXRlOjp5ZWFyKGx1YnJpZGF0ZTo6bm93KCkpYCINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDoNCiAgICBjb2RlX2Rvd25sb2FkOiB0cnVlDQogICAgdGhlbWU6IHlldGkNCiAgICBoaWdobGlnaHQ6IHRhbmdvDQogIGh0bWxfbm90ZWJvb2s6DQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQ0KICAgIHRoZW1lOiB5ZXRpDQogICAgaGlnaGxpZ2h0OiB0YW5nbw0KICBiZWFtZXJfcHJlc2VudGF0aW9uOiBkZWZhdWx0DQogIHdvcmRfZG9jdW1lbnQ6IGRlZmF1bHQNCiAgaW9zbGlkZXNfcHJlc2VudGF0aW9uOiBkZWZhdWx0DQogIHNsaWR5X3ByZXNlbnRhdGlvbjogDQogICAgdGhlbWU6IHlldGkNCiAgICBoaWdobGlnaHQ6IHRhbmdvDQogIHBkZl9kb2N1bWVudDogZGVmYXVsdA0KLS0tDQoNCg0KDQoNCg0KW1ZpZXcgdGhlIGxhdGVzdCByZWNvcmRpbmcgb2YgdGhpcyBsZWN0dXJlXShodHRwczovL1ItUmVzb3VyY2VzLm1hc3NleS5hYy5uei92aWRlb3MvMjUxTDEzLm1wNCkNCjwhLS0tIERhdGEgaXMgb24NCmh0dHBzOi8vci1yZXNvdXJjZXMubWFzc2V5LmFjLm56L2RhdGEvMTYxMjUxLw0KLS0tPg0KDQpgYGB7ciBzZXR1cCwgcHVybD1GQUxTRSwgaW5jbHVkZT1GQUxTRX0NCmxpYnJhcnkoa25pdHIpDQpvcHRzX2NodW5rJHNldChkZXY9YygicG5nIiwgInBkZiIpKQ0Kb3B0c19jaHVuayRzZXQoZmlnLmhlaWdodD02LCBmaWcud2lkdGg9NywgZmlnLnBhdGg9IkZpZ3VyZXMvIiwgZmlnLmFsdD0idW5sYWJlbGxlZCIpDQpvcHRzX2NodW5rJHNldChjb21tZW50PSIiLCBmaWcuYWxpZ249ImNlbnRlciIsIHRpZHk9VFJVRSkNCm9wdGlvbnMoa25pdHIua2FibGUuTkEgPSAnJykNCmxpYnJhcnkodGlkeXZlcnNlKQ0KbGlicmFyeShicm9vbSkNCmBgYA0KDQoNCjwhLS0tIERvIG5vdCBlZGl0IGFueXRoaW5nIGFib3ZlIHRoaXMgbGluZS4gLS0tPg0KDQpMaW5lYXIgcmVncmVzc2lvbiBtb2RlbHMgY2FuIGJlIGNvbnZlbmllbnRseSBleHByZXNzZWQgdXNpbmcgbWF0cml4ICBub3RhdGlvbi4NCg0KSW4gdGhpcyBsZWN0dXJlLCB3ZSB3aWxsIHNlZSBob3cgcmVzdWx0cyBmb3IgbGluZWFyIG1vZGVscyBhcmUgbXVjaCBtb3JlIGVhc2lseSBkZXJpdmVkIGFuZCB1bmRlcnN0b29kIHVzaW5nIG1hdHJpeCBub3RhdGlvbiB0aGFuIHdpdGhvdXQgaXQuDQoNCkFsc28gbm90ZSB0aGF0IHRoZSBtYXRyaXggYXBwcm9hY2ggaXMgd2hhdCBpcyBiZWluZyBkb25lIGluIHRoZSBiYWNrZ3JvdW5kIGJ5IGFsbCBnb29kIHN0YXRpc3RpY2FsIHNvZnR3YXJlIGluY2x1ZGluZyBSLg0KDQojIyBNYXRyaXggRm9ybXVsYXRpb24gb2YgdGhlIExpbmVhciBNb2RlbA0KDQokJFxib2xkc3ltYm9se3l9ID0gWCB7XGJvbGRzeW1ib2x7XGJldGF9fSArIHtcYm9sZHN5bWJvbHtcdmFyZXBzaWxvbn19DQpcbGFiZWx7ZXE6bWF0cml4TE19JCQNCg0Kd2hlcmUgJFxib2xkc3ltYm9se3l9JCBpcyB0aGUgcmVzcG9uc2UgdmVjdG9yLCAqWCogaXMgdGhlICoqbW9kZWwgbWF0cml4KiosICR7XGJvbGRzeW1ib2x7XGJldGF9fSQgaXMgdGhlIHZlY3RvciBvZiAqcCsxKiByZWdyZXNzaW9uIHBhcmFtZXRlcnMsIGFuZCAke1xib2xkc3ltYm9se1x2YXJlcHNpbG9ufX0kIGlzIHRoZSB2ZWN0b3Igb2YgKm4qIGVycm9yIHRlcm1zLg0KDQokXGJvbGRzeW1ib2x7eX0gPSBcbGVmdCBbDQpcYmVnaW57YXJyYXl9e2N9DQp5XzFcXA0KeV8yXFwNClx2ZG90c1xcDQp5X24NClxlbmR7YXJyYXl9IFxyaWdodCBdJCwgJFggPSBcbGVmdCBbIFxiZWdpbnthcnJheX17Y2NjY30NCjEgJiB4X3sxMX0gJiBcbGRvdHMgJiB4X3sxcH1cXA0KMSAmIHhfezIxfSAmIFxsZG90cyAmIHhfezJwfVxcDQpcdmRvdHMgJiBcdmRvdHMgJiBcZGRvdHMgJiBcdmRvdHNcXA0KMSAmIHhfe24xfSAmIFxsZG90cyAmIHhfe25wfQ0KXGVuZHthcnJheX0gXHJpZ2h0IF0kLCAkXGJvbGRzeW1ib2x7XGJldGF9ID0gXGxlZnQgWw0KXGJlZ2lue2FycmF5fXtjfQ0KXGJldGFfMFxcDQpcYmV0YV8xXFwNClx2ZG90c1xcDQpcYmV0YV9wXFwNClxlbmR7YXJyYXl9IFxyaWdodCBdJCwgYW5kICRcYm9sZHN5bWJvbHtcdmFyZXBzaWxvbn0gPSBcbGVmdCBbDQpcYmVnaW57YXJyYXl9e2N9DQpcdmFyZXBzaWxvbl8xXFwNClx2YXJlcHNpbG9uXzJcXA0KXHZkb3RzXFwNClx2YXJlcHNpbG9uX25cXA0KXGVuZHthcnJheX0NClxyaWdodCBdJA0KDQoNClRoZSBtZWFuIChleHBlY3RlZCkgdmFsdWUgb2YgdGhlIHJhbmRvbSB2ZWN0b3IgJFxib2xkc3ltYm9se3l9JCBpcw0KICAgICQkXGJlZ2lue2FsaWduZWR9DQogICAgXGJvbGRzeW1ib2x7XG11fSAmPSYgRVtcYm9sZHN5bWJvbHt5fV1cXA0KICAgICAmPSYgXGxlZnQgWyBcYmVnaW57YXJyYXl9e2N9IA0KICAgIEVbeV8xXSBcXCBFW3lfMl0gXFwgXHZkb3RzIFxcIEVbeV9uXSBcZW5ke2FycmF5fSBccmlnaHQgXSBcXA0KICAgICAmPSYgRVtYIHtcYm9sZHN5bWJvbHtcYmV0YX19ICsge1xib2xkc3ltYm9se1x2YXJlcHNpbG9ufX1dXFwgJj0mIFgge1xib2xkc3ltYm9se1xiZXRhfX1cZW5ke2FsaWduZWR9JCQNCg0KICAtIEhlcmUgd2UgaGF2ZSB1c2VkIHRoZSByZXN1bHQgdGhhdCBmb3IgYSByYW5kb20gdmVjdG9yICRcYm9sZHN5bWJvbHt6fSQsIGENCiAgICBtYXRyaXggKk0qIGFuZCBhIChub25yYW5kb20pIHZlY3RvciAkXGJvbGRzeW1ib2x7YX0kDQogICAgJCRFW1xib2xkc3ltYm9se2F9ICsgTVxib2xkc3ltYm9se3p9XSA9IFxib2xkc3ltYm9se2F9ICsgTSBFW1xib2xkc3ltYm9se3p9XSQkDQoNCiMjIyBMZWFzdCBTcXVhcmVzIEVzdGltYXRpb24gYnkgTWF0cmljZXMNCg0KRm9yIG9ic2VydmVkIHJlc3BvbnNlcyAkXGJvbGRzeW1ib2x7eX0kIHRoZSBzdW0gb2Ygc3F1YXJlZCBlcnJvcnMgY2FuIGJlDQogICAgd3JpdHRlbiBhcyAkJFNTKHtcYm9sZHN5bWJvbHtcYmV0YX19KSA9IChcYm9sZHN5bWJvbHt5fSAtIFgge1xib2xkc3ltYm9se1xiZXRhfX0pXlQgKFxib2xkc3ltYm9se3l9IC0gWCB7XGJvbGRzeW1ib2x7XGJldGF9fSkuJCQNCg0KICAtIFRoaXMgY2FuIGJlIG1pbmltaXNlZCB1c2luZyBtdWx0aXZhcmlhdGUgKHZlY3RvcikgY2FsY3VsdXMgdG8gZ2l2ZQ0KICAgIHRoZSBsZWFzdCBzcXVhcmVzIGVzdGltYXRlcyBhcw0KICAgICQkXGhhdHt7XGJvbGRzeW1ib2x7XGJldGF9fX0gPSAoWF5UIFgpXnstMX0gWF5UIFxib2xkc3ltYm9se3l9LiQkDQoNCiMjIyBIb3cgZGlkIHRoYXQgbWF0cml4IHNvbHV0aW9uIGNvbWUgYWJvdXQ/DQoNClN0YXJ0IHdpdGggJCRYIFxoYXR7e1xib2xkc3ltYm9se1xiZXRhfX19ID1cYm9sZHN5bWJvbHt5fSQkIGFuZCBkbyBhIHNlcmllcyBvZiBwcmUtbXVsdGlwbGljYXRpb25zOiBGaXJzdCBieSAkWF5UJCAodGhlIHRyYW5zcG9zZSBvZiB0aGUgJFgkIG1vZGVsIG1hdHJpeC4NCiQkWF5UIFggXGhhdHt7XGJvbGRzeW1ib2x7XGJldGF9fX0gPSBYXlQgXGJvbGRzeW1ib2x7eX0kJCwgdGhlbiB0aGUgaW52ZXJzZSBvZiAkKFheVCBYKSQgdG8gZ2l2ZQ0KJCQoWF5UIFgpXnstMX0gKFheVCBYKSBcaGF0e3tcYm9sZHN5bWJvbHtcYmV0YX19fSA9IChYXlQgWCleey0xfSBYXlQgXGJvbGRzeW1ib2x7eX0kJC4NCg0KQW55dGhpbmcgbXVsdGlwbGllZCBieSBpdHMgaW52ZXJzZSBpcyBlcXVhbCB0byB0aGUgaWRlbnRpdHkgbWF0cml4LCBhbmQgY2FuY2VscyBpdHNlbGYgb3V0LCBycmVkdWNpbmcgdGhlIGxlZnQgaGFuZCBzaWRlIHRvIGJlIHRoZSBwYXJhbWV0ZXIgZXN0aW1hdGVzICRcaGF0e3tcYm9sZHN5bWJvbHtcYmV0YX19fSAkIHdlIHNlZWsuDQoNCiMjIyBTaW1wbGUgTGluZWFyIFJlZ3Jlc3Npb24gaW4gbWF0cml4IGZvcm0NCg0KRm9yIGEgc2ltcGxlIGxpbmVhciByZWdyZXNzaW9uIG1vZGVsIHRoZSBtb2RlbCBtYXRyaXggaXMNCiQkWCA9IFxsZWZ0IFsNClxiZWdpbnthcnJheX17Y2N9DQoxICYgeF97MX0gXFwNCjEgJiB4X3syfSBcXA0KXHZkb3RzICYgXHZkb3RzIFxcDQoxICYgeF97bn0NClxlbmR7YXJyYXl9DQpccmlnaHQgXSQkIElmIHdlIG9ic2VydmUgcmVzcG9uc2VzICRcYm9sZHN5bWJvbHt5fSQsIHRoZSBsZWFzdCBzcXVhcmVzIGVzdGltYXRlDQpvZiAke1xib2xkc3ltYm9se1xiZXRhfX0kIGlzOiAkJFxiZWdpbnthbGlnbmVkfQ0KXGhhdHtcYm9sZHN5bWJvbHtcYmV0YX19ICY9JiAoWF5UIFgpXnstMX0gWF5UIFxib2xkc3ltYm9se3l9ID0gDQpcbGVmdCBbDQpcYmVnaW57YXJyYXl9e2NjfQ0KbiAmIG4gXGJhcnt4fVxcDQpuXGJhcnt4fSAmIFxzdW1faSB4X2leMiANClxlbmR7YXJyYXl9DQpccmlnaHQgXV57LTF9IA0KXGxlZnQgWw0KXGJlZ2lue2FycmF5fXtjY2NjfQ0KMSAmIDEgJiBcbGRvdHMgJiAxIFxcDQp4XzEgJiB4XzIgJiBcbGRvdHMgJiB4X24gDQpcZW5ke2FycmF5fQ0KXHJpZ2h0IF0NClxsZWZ0IFsNClxiZWdpbnthcnJheX17Y30NCnlfMVxcDQp5XzJcXA0KXHZkb3RzXFwNCnlfbg0KXGVuZHthcnJheX0NClxyaWdodCBdIFxcDQogJj0mDQpcZnJhY3sxfXtuIHNfe3h4fX0gXGxlZnQgWw0KXGJlZ2lue2FycmF5fXtjY30NClxzdW1faSB4X2leMiAmIC0gbiBcYmFye3h9XFwNCi0gblxiYXJ7eH0gJiBuIA0KXGVuZHthcnJheX0NClxyaWdodCBdIA0KXGxlZnQgWyANClxiZWdpbnthcnJheX17Y30NCm4gXGJhcnt5fVxcDQpcc3VtX2kgeF9pIHlfaQ0KXGVuZHthcnJheX0NClxyaWdodCBdDQo9IA0KXGZyYWN7MX17c197eHh9fSBcbGVmdCBbIA0KXGJlZ2lue2FycmF5fXtjfQ0KXGJhcnt5fSBcc3VtX2kgeF9pXjIgLSBcYmFye3h9IFxzdW1faXt4X2kgeV9pfSBcXA0Kc197eHl9DQpcZW5ke2FycmF5fQ0KXHJpZ2h0IF1cZW5ke2FsaWduZWR9JCQNCg0KIyMgUHJlZGljdGlvbiBhbmQgVGhlIEhhdCBNYXRyaXgNCg0KVGhlIHZlY3RvciBvZiBmaXR0ZWQgdmFsdWVzIGlzIGdpdmVuIGJ5ICRcaGF0e1xib2xkc3ltYm9se1xtdX19ID0gWCBcaGF0e1xib2xkc3ltYm9se1xiZXRhfX0gPSBYIChYXlRYKV57LTF9IFheVCBcYm9sZHN5bWJvbHt5fSQgYW5kIHRoZSB2ZWN0b3Igb2YgcmVzaWR1YWxzIGJ5ICRcYm9sZHN5bWJvbHtlfSA9IFxib2xkc3ltYm9se3l9IC0gXGhhdHtcYm9sZHN5bWJvbHtcbXV9fSQuDQoNCg0KVGhlIGVxdWF0aW9uIGZvciB0aGUgZml0dGVkIHZhbHVlcyBqdXN0IGdpdmVuLCBjYW4gYmUgcmUtd3JpdHRlbiAkXGhhdHtcYm9sZHN5bWJvbHtcbXV9fSA9IEggXGJvbGRzeW1ib2x7eX0gPSBcaGF0e1xib2xkc3ltYm9se3l9fSQgIHdoZXJlICpIID0gWCAoWF5UXlgpXi0xXlheVF4qIGlzDQogICAgb2Z0ZW4gY2FsbGVkIHRoZSAqKmhhdCBtYXRyaXgqKiBiZWNhdXNlIGl0ICJwdXRzIGhhdHMgb24gdGhpbmdzIiENCg0KV2UgaGF2ZSB0aGUgZXF1YWxpdHkgJFxoYXR7XGJvbGRzeW1ib2x7eX19ID0gXGhhdHtcYm9sZHN5bWJvbHtcbXV9fSQgYmVjYXVzZSB3ZSB1c2UgdGhlDQogICAgc2FtZSB2YWx1ZXMgZm9yIHByZWRpY3Rpb24gYW5kIG1lYW4gZXN0aW1hdGlvbi4NCg0KIyMgQ292YXJpYW5jZSBNYXRyaWNlcw0KDQogIC0gVGhlIHZhcmlhbmNlLWNvdmFyaWFuY2UgKG9yIHNpbXBsZSBjb3ZhcmlhbmNlLCBvciBkaXNwZXJzaW9uIG1hdHJpeCkNCiAgICBoYXMgdmFyaWFuY2VzIGRvd24gdGhlIGRpYWdvbmFsIGFuZCBjb3ZhcmlhbmNlcyBvZmYgdGhlIGRpYWdvbmFsLg0KDQogIC0gSXQgY2FuIGJlIHNob3duIHRoYXQgZm9yIGEgbWF0cml4ICpNKiBhbmQgcmFuZG9tIHZlY3RvciAkXGJvbGRzeW1ib2x7en0kDQogICAgKG9mIGFwcHJvcHJpYXRlIGRpbWVuc2lvbnMpDQogICAgJCRcbWJveHtWYXJ9IChNXGJvbGRzeW1ib2x7en0pID0gTSBcbWJveHtWYXJ9IChcYm9sZHN5bWJvbHt6fSkgTV5ULiQkDQoNCiMjIFRoZSBDb3ZhcmlhbmNlIE1hdHJpeCBmb3IgJFxoYXR7XGJvbGRzeW1ib2x7XGJldGF9fSQNCg0KVGhlIGNvdmFyaWFuY2UgbWF0cml4IG9mICRcaGF0e1xib2xkc3ltYm9se1xiZXRhfX0kIGlzIA0KDQokJFxiZWdpbnthbGlnbmVkfQ0KICAgIFxtYm94e1Zhcn0oXGhhdHtcYm9sZHN5bWJvbHtcYmV0YX19KSA9IFxtYm94e1Zhcn1bIChYXlRYKV57LTF9IFheVCBcYm9sZHN5bWJvbHt5fSBdICY9JiAoWF5UWCleey0xfSBYXlQgXG1ib3h7VmFyfShcYm9sZHN5bWJvbHt5fSkgWyhYXlRYKV57LTF9IFheVF1eVFxcDQogICAgICAgICY9JiAoWF5UWCleey0xfSBYXlQgXHNpZ21hXjIgSSBbKFheVFgpXnstMX0gWF5UXV5UXFwNCiAgICAgICAgJj0mIFxzaWdtYV4yIChYXlRYKV57LTF9IFheVFggKFheVCBYKV57LTF9XFwNCiAgICAgICAgJj0mIFxzaWdtYV4yIChYXlRYKV57LTF9XGVuZHthbGlnbmVkfSQkDQoNCldlIGNhbiB1c2UgJFxtYm94e1Zhcn0oXGJvbGRzeW1ib2x7eX0pID0gXHNpZ21hXjIgSSQgc2luY2UgdGhlIHJlc3BvbnNlcyBhcmUNCiAgICBpbmRlcGVuZGVudCBhbmQgaGVuY2UgdW5jb3JyZWxhdGVkLg0KDQoNCi0gVGhlIHZhcmlhbmNlcyBleHByZXNzIHRoZSB2YXJpYWJpbGl0eSBvZiB0aGUgZXN0aW1hdG9ycyBmcm9tIHNhbXBsZQ0KICAgIHRvIHNhbXBsZS4NCg0KLSBUaGUgY292YXJpYW5jZXMgZGVzY3JpYmUgdGhlIGludGVyLWRlcGVuZGVuY2Ugb2YgZXN0aW1hdG9ycy4NCg0KDQpUaGUgbGVhZGluZyBkaWFnb25hbCBvZiB0aGlzIG1hdHJpeCBpcyB0aGUgYmFzaXMgZm9yIHRoZSBzdGFuZGFyZCBlcnJvcnMgb2Ygb3VyIHBhcmFtZXRlciBlc3RpbWF0ZXMgc2VlbiBpbiBvdXIgcmVncmVzc2lvbiBvdXRwdXQuDQppcyA=