Mean squared error and squared bias
Back

Mean squared error. The so-called theory of Mallows C_p is concerned the theoretical properties of mean squared error and squared bias. We summarize the results here. The theoretical expected values of the response values are Xβ, but we are only using ŷ_A=X₁b₁. X₁ is the part of X that we are using and b₁ is our estimate of the parameters, β, when A terms are used in the decomposition of X. The estimated response value for the i^th response value is ŷ_i=b₁^Txⁱ. The residual sums of squares, RSS_A, is given by

                   RSS_A = |y - ŷ_A|² = |y₁ - b₁^Tx¹|² + … + |y_N - b₁^Tx^N|².

 It has the mean value

(1)               E(RSS_A) = (N-A)σ² + g₂^TD₂g₂

The squared bias, J_A= |X₁b₁- Xβ|² has the mean value

(2)               E(J_A) = A σ² + [g_A+1² (t_A+1^Tt_A+1) + … + g_K² (t_K^Tt_K)] = Aσ² + g₂^TD₂g₂

The term g₂^TD₂g₂ is estimated by c₂^TD₂c₂. Eliminating the term g₂^TD₂g₂ gives

(3)               E(J_A)/σ² = E(RSS_A)/σ² – N + 2A.

This equation leads to the so-called Mallows C_p-criterion that is given by

(4)               C_A = RSS_A/s² - N + 2A,

C_A is an estimate of E(J_A)/σ². Here s² is a ‘good’ estimate of σ². These equations are important from methodological point of view:

i) (2) shows that it is important to keep the dimension as low as possible

ii) The amount of squared bias, c₂^TD₂c₂, that should be allowed, can be found as a trade-off from (1) or (2).

iii) (2) and (4) show that C_A should be as small as possible and as close to A as possible.

iv) The penalty of of one dimension extra in the model is approximately 2σ².

As pointed out by Chatterjee¹ the results i)-iv) are valid in general for both linear and non-linear models, and independently of the question of overfitting  or underfitting. Unfortunately, these formulae cannot be used to identify the dimension of the model. The reason is that they only are concerned with the fit of the given model. It is instructive to look at an example.

In typical application a certain maximal value for the dimension is chosen. The residual variance at that dimension is chosen as an estimate of the variance σ².

In Fig. 1 the values of C_A in equation (4) are shown for the furnace data. The dimension for these data is 5 or 6. The minimal value of C_A is obtained at dimension 12. (The results would be the same if e.g, the maximal dimension was chosen as 15).

The general experience is that the application of Mallows C_p-value leads to overfitting of the data.  The reason is that the measures are very week. This can be seen by looking at (2). The modelling should stop if E(J_A) < E(J_A+1), or equivalently if g_A+1² (t_A+1^Tt_A+1) < σ² . The F-value for testing the significance of the regression coefficient c_A+1 is F=c_A+1² (t_A+1^Tt_A+1)/s². Thus the modeling should stop, if the F-value is less than 1, F<1.

Back