ISLP Notes - Chapter 2
The essence of statistical learning is crafting functions that predict or classify things about the world based on a set of inputs.
How do we craft these functions? You start with one of two options.
-
Make an assumption on the form of your function. Try assuming its a straight line, and then use a procedure to calculate the best slope for that line.
-
Make no assumptions of the data. Algorithms like K nearest neighbors just observe the data nearby to make their predictions.
How do we know our function is a good? Training Data. If we have data with known outputs, we can compare the difference between our function’s answer and the known output. A common algorithm for this is mean squared error.
MSE is not without limitations. You may have a low error rate against your training data, only to find out the error rate is high in the real world. This is referred to as being overfit.
Functions with high MSE tend to stem from two factors:
- Variance. Does your error rate change when you swap out your training set for another one? Your model may be too flexible.
- Bias. Straight lines have a terrible time acting as estimates for parabolas. Your function will be too rigid for such a problem.
An important trade off arises.
- Rigid models will have less variance, and more bias.
- Flexible models will have more variance, and less bias.