Significance testing and the prediction variance
Back
Here we shall consider the prediction variances of new samples. A standard linear regression model, y~N(Xb,s²I), is assumed. Furthermore, we assume that we have selected A components. The response value of the new sample, x₀, is estimated as y(x₀)=b₁^Tx₀. The variance of y(x₀) is given by 

                       Var(y(x₀)) =  s² x₀^T (X₁^TX₁)⁺ x₀ 

@ [(y^Ty)- {(y^Tt₁)²/(t₁^Tt₁)+…+(y^Tt_A)²/(t_A^Tt_A)}]/(N-A) 

´ [(x₀^Tr₁)²/(t₁^Tt₁) + … + (x₀^Tr_A)²/(t_A^Tt_A)] 

Here we have neglected the possible bias. The residual variance is given by

            s²_A = [(y^Ty)- {(y^Tt₁)²/(t₁^Tt₁)+…+(y^Tt_A)²/(t_A^Tt_A)}]/(N-A)

Here it is assumed that X has been decomposed as X=t₁p₁^T+...+t_Ap_A^T+ X₀. We have R=(P^T)^-1. If the task is variable selection, the matrix P will be a lower diagonal matrix. When we are finding the next score vector, t_A+1, we have two aspects to take into consideration: 

i) Reduction in fit:                        (y^Tt_A+1)²/(t_A+1^Tt_A+1)
ii) Increase in model variation:     (x₀^Tr_A+1)²/(t_A+1^Tt_A+1)

When handling these two terms, we may consider the term (x₀^Tr_A+1)² as a constant even though r_A+1 will depend on the score vector, t_A+1, we are trying to find. The reason is that r_A+1 typically has length around one. Also, the value of (x₀^Tr_A+1) will vary with new data, x₀. The H-principle suggests considering the product of i) and the inverse of ii). In this way we find an optimal balance between the improvement in fit and the associated increase in the model variation. The procedure to obtain an optimal balance can be based on other measures of reduction in fit and also other measures of increase in model variation. The main issue is to take the increase in model variation into account, when finding new score vectors. Furthermore, we have considered here a standard linear regression model. Other mathematical models will suggest different measures of fit (improvement of the modeling criterion) and measures of prediction associated with the solution at each step.

Significance testing. Traditional significance testing is only concerned with the question of fit. We shall look at the procedure closer. The regression coefficient associated with a new score vector t_A+1 is c_A+1=(y^Tt_A+1)/(t_A+1^Tt_A+1). It significance is compared to its approximate variance, s²_A+1/(t_A+1^Tt_A+1), by an F-statistic, 

           F = [(y^Tt_A+1)/(t_A+1^Tt_A+1)]²/[ s²_A+1/(t_A+1^Tt_A+1)]. 

There is a useful way to look at this test by considering the residual values of the response variable. Let u₀=y and define 

                       u_i= u_i-1 - (y^Tt_i)/(t_i^Tt_i) t_i,            i=1,2, …, A.

With this notation u_i is the residual after i steps and using (y^Tt_A+1)=(u_A^Tt_A+1), we can write

s²_A+1 = [(u_A^Tu_A)- (u_A^Tt_A+1)²/(t_A+1^Tt_A+1)]/(N-A-1)

 This gives

                    F = [(u_A^Tt_A+1)²/(t_A+1^Tt_A+1)]/[(u_A^Tu_A)- (u_A^Tt_A+1)²/(t_A+1^Tt_A+1)]/(N-A-1)

                   = [r(u_A, t_A+1)]²/(1-[r(u_A, t_A+1)]²)/(N-A-1).

 Here [r(u_A, t_A+1)]² is the squared simple correlation coefficient between the residual of the response variable after A components, u_A, and the next score vector, t_A+1. We see that F is a monotone function of the correlation coefficient. Thus the F-test is equivalent to test the significance of the correlation coefficient between the present residual response values and the next score vector. The correlation coefficient is invariant to the size of the score vector, t_A+1. Thus, significance testing is only concerned with improvement in fit, i) above. This is unfortunate, because the terms in ii) are stochastically independent of the terms in i). In practice it means that the terms in ii) can be large or small. Thus significance testing does not provide with information if the prediction variance, Var(y(x₀)), is large or small. Program packages using significance testing to find variables or dimensions generally lead to results that represent overfitting of the data in the sense that the prediction variance is too large. The reason is partly that e.g., 5% level of significance is used, but should be much lower than 5% because the methods are searching in data. Another reason is that, as mentioned above, significance testing does not pay any consideration to the model variation.  

Back