Cross-validation

Picking estimators¶

You are given a dataset with $n=10$ points. Using this data, you hope to learn how to guess a binary response $Y$ from input features $X$. You have been studying machine learning in-depth and have learned about 17 completely different approaches for this kind of classification problem. You assess the quality of each of these estimators using 5-fold cross-validation and learn that the fourth estimator seems to achieve the best performance. You then gain access to another dataset with 10 more points. You may assume the new dataset is drawn from the same distribution as the first. You then train each of the 17 estimators on the first dataset and evaluate it on the new dataset. True or false: in most cases, the fourth estimator will perform better than all others on the new held-out data.

Picking estimators, II¶

You are given a dataset with $n=1000$ points concerning many predictors and one numerical response. You consider two estimators: ridge-regularized linear regression and LASSO-regularized linear regression. For both methods, assume that the regularization strengths are chosen by cross-validation. To see which approach is better, you try 5-fold cross-validation (note that this is a form of nested cross-validation, since the estimators themselves use cross-validation to pick regularization strength). The results suggest that LASSO is a bit better. You then decide to try LOOCV instead. With LOOCV it appears that LASSO is a bit worse. Which would be the more appropriate interpretation of these findings?

1. LASSO is a bit worse (LOOCV gives a more reliable assessment, though it has the disadvantage of being more computationally intensive)

2. LASSO is a bit better (LOOCV gives a less reliable assessment, though it has the advantage of giving the same result every time because it does not require random resampling)

3. Can’t tell.

Picking regularization strength¶

You are given a dataset with $n=15$ points. Using this data, you hope to learn how to guess a binary response $Y$ from input features $X$. You know you want to use ridge-regularized logistic regression, but you’re not sure what value of λ to use. Your plan is to select a grid of values for λ, use cross-validation to assess the quality of the estimator based on each value, and then select the best. True or false: from a statistical standpoint, it is a better idea to consider only relatively few possible values of λ, e.g. $\lambda \in \{.01,.1,1.0,10,100\}$. Considering too many choices for λ leads to a kind of over-fitting.

LOOCV by hand¶

Consider a dataset concerning a single numerical response $Y \in \mathbb{R}$. The dataset has no predictors. Here is the dataset.

   Sample ID   Response
  ----------- ----------
      #1         2.0
      #2         4.0
      #3         3.0

We seek to estimate the regression function. In this degenerate case, the regression function is simply the expected value of $Y$, $\mu = \mathbb{E}[Y]$. For our estimator, we will use the sample average. For example, applying this estimator to the full dataset, we obtain $\hat \mu =(2+4+3)/3=3$.

In this problem, you will perform LOOCV by hand to estimate $\mathbb{E}[(Y-\hat \mu)^2]$. For each test-train split, use the sample average of the training samples to estimate μ and evaluate the squared difference between your estimate and the held-out test sample. Your final answer will be the average of these squared differences. Your final answer does not need to be written in a simplified form (however, here as elsewhere your answer must must be highly legible!).