Skip to article frontmatterSkip to article content

Standardization

How does one apply the standard scaler (also known as “standardization”)?

  1. First calculate the median and interquartile range of the predictor values. Then subtract off the median from the predictor values. Then divide by interquartile range.

  2. First calculate the mean and standard deviation of the predictor values. Then subtract off the mean from the predictor values. Then divide by standard deviation.

  3. First calculate the min and max of the predictor values. Then subtract off the min from the predictor values. Then divide by the range (i.e. max minus min).

Solutions

The standard scaler uses the second method. But, as an aside, the other two are actually fine too. The second one is called “RobustScaler” in sklearn and the third one is called “MinMaxScaler.” The interested student might look up https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing to see other options as well. Note: for this class, you don’t need to know about any of these methods; you only need to know about the standard scaler.

Standardization using all available data

True or false: when using the standard scaler (also known as “standardization”), it is considered best practice to fit the mean and standard deviation only once, using all available data. This is true even if you will be making test/train splits later for the purposes of assessing performance.

Solutions

False. In most circles, it is considered best practice is to include the standardization process as part of the estimator and fit it only on training data if you are making test/train splits. That’s why sklearn has this “pipeline” architecture. Doing so means that every time you fit the estimator on some subset of the data (in the context of a test/train split or cross-validation) you end up refitting the mean and standard deviation.

Standardization with LDA

True or false: assuming you could use a computer with infinite numerical precision, the predictions made by LDA based on the raw features are the same as the predictions of LDA based on rescaled features.

Solutions

True! LDA is invariant to rescaling. Same idea goes with QDA, logistic regression (assuming you don’t use any ridge or LASSO penalties) and linear regression (assuming you don’t use any penalties). By contrast, if you use KNN or any sort of ridge/LASSO penalties, your predictions may indeed change with rescaling. Note: the condition about infinite numerical precision is necessary because in practice scaling can help with round-off errors (even for LDA, QDA, etc).

Standardization in linear regression

True or false: assuming you could use a computer with infinite numerical precision, assuming you do not use any ridge or LASSO penalties, the coefficients β estimated by liner regression will be the same regardless of whether you preprocess your data by scaling.

Solutions

False! Rescaling will change the estimated coefficients. But it won’t change the predictions. Here’s an example of how this works, in the sample case with one raw feature.

  1. Say that you train linear regression on the unscaled data and you get coefficients β0,β1\beta_{0},\beta_{1}. Then, given a new test query xx you’ll predict β0+β1x\beta_{0}+\beta_{1}x.

  2. If you preprocess the data by multiplying the predictors by α, assuming infinite numerical precision, you’ll get coefficients β~0=β0,β~1=β1/α\tilde{\beta}_{0}=\beta_{0},\tilde{\beta}_{1}=\beta_{1}/\alpha. Then, given a new test query xx, you’ll obtain the transformed version x~=xα\tilde{x}=x\alpha and apply the rule β~0+β~1x~=β0+(β1/α)xα=β0+β1α\tilde{\beta}_{0}+\tilde{\beta}_{1}\tilde{x}=\beta_{0}+\left(\beta_{1}/\alpha\right)x\alpha=\beta_{0}+\beta_{1}\alpha.

Thus for linear regression without penalties, the predictions won’t be affected by rescaling but the coefficients will.

Standardization with nonlinear featurization

Given a dataset ((X1,Y1),…,(Xn,Yn))\left(\left(X_{1},Y_{1}\right),\ldots,\left(X_{n},Y_{n}\right)\right) with p=1p=1 predictor and one numerical response, consider the following procedure for estimating the regression function.

β^=arg⁡min⁡β(∑i=1n(Yi−β0−β1eXi))f^(x;β^)=β0+β1cos⁡x\begin{aligned} \hat{\beta} & =\arg\min_{\beta}\left(\sum_{i=1}^{n}(Y_{i}-\beta_{0}-\beta_{1}e^{X_{i}})\right)\\ \hat{f}(x;\hat{\beta}) & =\beta_{0}+\beta_{1}\cos x \end{aligned}
(1)

Will this estimator lead to different predictions if you rescale the data?

Solutions

Yes. There are some featurizations, such as this one, where a least square approach will be affected by scaling.

Rescaling with quartic coefficient penalties

Given a dataset ((X1,Y1),…,(Xn,Yn))\left(\left(X_{1},Y_{1}\right),\ldots,\left(X_{n},Y_{n}\right)\right) with p=1p=1 predictor and one numerical response, consider the following procedure for estimating the regression function.

β^=arg⁡min⁡β(∑i=1n(Yi−β0−β1Xi)2+β4)f^(x;β)=β0+β1x\begin{aligned} \hat{\beta} & =\arg\min_{\beta}\left(\sum_{i=1}^{n}(Y_{i}-\beta_{0}-\beta_{1}X_{i})^{2}+\beta^{4}\right)\\ \hat{f}(x;\beta) & =\beta_{0}+\beta_{1}x \end{aligned}
(2)

Will this estimator lead to different predictions if you rescale the data?

Solutions

Yes. This estimator is much like a ridge regression estimator, except with β4\beta^{4} instead of β2\beta^{2}. As with ridge, this estimation procedure will be affected by scaling.

Forests and trees
Forests
Featurization, standardization, and the magic of splines
Featurization