Finding the minimizers of RSS

In chapter 3 of ISLP The authors introduce the minimizers of the residual sum of squares with respect to $\hat{B_0}$ and $\hat{B_1}$ for simple linear regression.

The least squares approach chooses $\hat{B_0}$ and $\hat{B_1}$ to minimize the RSS. Using some calculus, one can show that the minimizers are

$\hat{B_1} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2}$

$\hat{B_0} = \bar{y} - \hat{B_1}\bar{x}$

I’m going to walk through how these functions are derived from the RSS.

RSS can be thought of as a function producing the Z axis value in 3d space. It says: “given an intercept value plotted on the x axis, and a slope plotted on the y axis, I will give you the value on the z axis.” We can isolate the influence of slope and intercept by taking the partial derivatives. Once we’ve isolated that, we can solve them for 0 to arrive at the formulas presented in the book.

Why zero? The lowest RSS value in our 3d space sits at the bottom of a bowl. This has the convenient fact of meaning at the bottom of this bowl the slope of the X and Y axis are zero. Solving for the minimizers gives us the X and Y coordinates where that lowest Z coordinate is.

Taking the partial derivative of RSS with respect to $\hat{B_0}$

RSS is the squared difference between our sample data points and our estimates. It can be written as

$\sum_{i=1}^{n}(y_i - \hat{y_i})^2$

where $y_i$ is an observation in the data set, and $\hat{y_i}$ is an estimated value based on some Slope and Intercept for an input value $x$. $\hat{y_i}$ can be expanded to the function that produces its value. $\hat{\beta_0}+\hat{\beta_1}x$

$\sum_{i=1}^{n}(y_i - (\hat{\beta_0} + \hat{\beta_1}x_i))^2$

Then the negative sign can be distributed.

$\sum_{i=1}^{n}(y_i - \hat{\beta_0} - \hat{\beta_1}x_i)^2$

The Sum Rule means we can ignore the summation sign and focus on just solving $(y_i - \hat{\beta_0} - \hat{\beta_1}x_i)^2$. Since there is an inner term and outer term (squaring the inner term) we can use the chain rule to solve $y_i - \hat{\beta_0} - \hat{\beta_1}x_i$ first.

$y_i$ is a constant. set to 0
$\hat{\beta_i}x_i$ is a constant. set to 0 $-\hat{\beta_i}$ is a variable, set to -1

0 - 1 + 0 = -1

Apply this to the function in the chain rule.

$f’(g(x)) \times g’(x) = f’(g(x)) \times -1$

The outer function, which squares the the inner can be written as $f(u) = u^2$. This is then solved with the power rule. $f’(u) = 2u$. plug in g(x) for u

$2(y_i - \beta_0 - \beta_1 x_i) \times -1$

Simplify $2 \times -1$, and add back the summation sign. The -2 can be moved to the left of the sign.

$f_{\beta_0}^{\prime} = -2 \sum_{i=1}^{n}(y_i - \hat{\beta_0} - \hat{\beta_1} x_i)$

Taking the partial derivative of RSS with respect to $\hat{B_1}$

Follow the same steps we did as before, this time solving $y_i - \hat{\beta_0} - \hat{\beta_1}x_i$, with respect to $\beta_1$

Y is a constant. set to 0. $\hat{\beta_0}$ is a constant. set to 0. $-\hat{\beta_1}$ is the variable, set to -1. x is a multiplier of the variable, simplify $-1 \times x$ to $-x$

$0 - 0 - x = -x$

plug back in to the chain formula

$2(y_i - \hat{\beta_0} - \hat{\beta_1} x_i) \times -x$

simplify the multipliers 2 and -x

$-2x(y_i - \hat{\beta_0} - \hat{\beta_1} x_i)$

Add back the summation sign, again moving the -2 to the left.

$f_{\beta_1}^{\prime}=-2 \sum_{i=1}^{n}x(y_i - \beta_0 - \beta_1 x_i)$

Derive the minimizers by setting the derivatives to zero

With $\hat{\beta_0}$ plotted on the x axis and $\hat{\beta_1}$ on the Y. Solving the derivatives for zero gives the X, Y coordinates where Z is the smallest value. The smallest Z value will have a tangent plane with a slope of zero along both the X and Y axis.

The minimizer For $\hat{\beta_0}$

$-2 \sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i) = 0$

Divide each side by 0, -2 is canceled out.

$\sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i) = 0$

Distribute the summation sign between $y_i$, $\hat{\beta_0}$, and $\hat{\beta_1}x$

$\sum_{i=1}^{n}y_i - \sum_{i=1}^{n}\hat{\beta_0} - \sum_{i=1}^{n}\hat{\beta_1}x_i = 0$

For $\sum_{i=1}^{n}\hat{\beta_0}$ , $\hat{\beta_0}$ never changes for n. It is the sum of $\hat{\beta_0}$ n times. It can be rewritten as $n\hat{\beta_0}$

Similarly, $\hat{\beta_1}$ is constant in $\sum_{i=1}^{n}\hat{\beta_1}x_i$, it can moved to the left of the summation sign. $\hat{\beta_1}\sum_{i=1}^{n}x_i$

Our term is now $\sum_{i=1}^{n}y_i - n\hat{\beta_0} - \hat{\beta_1}\sum_{i=1}^{n}x_i = 0$

Isolate $\hat{\beta_0}$, first by adding $n\hat{\beta_0}$ to both sides

$\sum_{i=1}^{n}y_i - \hat{\beta_1}\sum_{i=1}^{n}x_i = n\hat{\beta_0}$

Then dividing n by both sides

$\frac{\sum_{i=1}^{n}y_i - \hat{\beta_1}\sum_{i=1}^{n}x_i}{n} = \hat{\beta_0}$

We can make another simplification here. The sum of terms Y and X divided by the total number of samples is the mean. $\sum_{i=1}^{n}y_i$ becomes $\bar{y}$ and $\sum_{i=1}^{n}x_i$ becomes $\bar{x}$. This yields the final formula as presented by the authors.

$\bar{y} - \hat{\beta_1}\bar{x} = \hat{\beta_0}$

The minimizer $\hat{\beta_1}$

$f_{\beta_1}^{\prime}=-2 \sum_{i=1}^{n}x(y_i - \beta_0 - \beta_1 x_i)$ = 0

divide out -2 from both sides.

$\sum_{i=1}^{n}x(y_i - \beta_0 - \beta_1 x_i)$ = 0

substitute $\hat{\beta_0}$ with the formula we solved for in the previous section.

$\sum_{i=1}^{n}x(y_i - (\bar{y} - \hat{\beta_1}\bar{x}) - \beta_1 x_i)$ = 0

distribute the negative sign.

$\sum_{i=1}^{n}x(y_i - \bar{y} + \hat{\beta_1}\bar{x} - \bar{\beta_1} x_i)$ = 0

$\hat{\beta_1}$ can be factored out of its terms.

$\sum_{i=1}^{n}x_i((y_i - \bar{y}) - \hat{\beta_1}(x_i - \bar{x}))$ = 0

x can be distributed in to the paraenthesis.

$\sum_{i=1}^{n}x_i(y_i - \bar{y}) - \hat{\beta_1}x_i(x_i - \bar{x})$ = 0

The summation sign can be split between the two groups.

$\sum_{i=1}^{n}x_i(y_i - \bar{y}) - \sum_{i=1}^{n}\hat{\beta_1}x_i(x_i - \bar{x})$ = 0

Since $\hat{\beta_1}$ is a constant, we can put it to the left of the summation sign.

$\sum_{i=1}^{n}x_i(y_i - \bar{y}) - \hat{\beta_1}\sum_{i=1}^{n}x_i(x_i - \bar{x})$ = 0

Add the second summation term to both sides.

$\sum_{i=1}^{n}x_i(y_i - \bar{y}) = \hat{\beta_1}\sum_{i=1}^{n}x_i(x_i - \bar{x})$

Divide the right side term by both sides to solve for B1.

$\frac{\sum_{i=1}^{n}x_i(y_i - \bar{y})}{\sum_{i=1}^{n}x_i(x_i - \bar{x})} = \hat{\beta_1}$

we can show that the numerator $x_i(y_i - \hat{y}) = (x_i - \bar{x})(y_i-\bar{y})$:

First, we expand the right hand side: $\sum_{i}^{n}x_i(y_i-\bar{y}) - \bar{x}(y_i-\bar{y})$

and split the summation sign.

$\sum x_i(y_i - \bar{y}) - \bar{x}\sum(y_i - \bar{y})$

and because $\sum(y_i - \bar{y})=0$. We are left with $x_i(y_i - \bar{y})$.

For the denominator, we can employ a trick to rewrite $x_i = (x_i - \bar{x}) + \bar{x}$

$\sum_{i}^{n}x_i(x_i-\bar{x}) = \sum_{i}^{n}((x_i-\bar{x}) + \hat{x})(x_i - \bar{x})$

and then distribute $x_i - \bar{x}$

$((x - \bar{x}) + \bar{x})(x_i - \bar{x}) = (x_i - \bar{x})(x_i - \bar{x}) + \bar{x}(x_i - \bar{x})$

which simplifies to

$\sum(x_i - \bar{x})^2 + \bar{x}\sum(x_i-\bar{x})$

and just like we saw in the numerator, $\sum(x_i - \bar{x}) = 0$

leaving us with the formula presented in the book.

$\frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i- \bar{x})^2} = \hat{\beta_1}$