The uncertainties of predictions in linear regression

 Back

In industry the primary interest is in the uncertainties of predictions associated with new samples. We shall consider closer the parts that enter the prediction variance, when assuming a standard linear regression model, y~N(Xb,s²I), where X is an N´K design matrix. The variance of the estimated response, y(x₀)=b^Tx₀, associated with a new sample x₀ is given by

           Var(y(x₀)) @ [(y^Ty) - y^T X(X^TX)^-1 X^Ty ][1+x₀^T (X^TX)^-1 x₀]/(N-K)

When expanding the model by adding new components/variables, one can show that the measures

i)            the error of fit, [(y^Ty) - y^T X(X^TX)^-1 X^Ty ], always decreases

ii)          the model variation, (1+x₀^T (X^TX)^-1 x₀), always increases

(In theory these measures may be unchanged, when adding new components, but in practice these changes always occur).

From a modelling point of view the following considerations are important. If the data values (y X) follow a multivariate normal distribution, then a) and b) given by

a)     the error of fit, [(y^Ty) - y^T X(X^TX)^-1 X^Ty ]

b)    the model precision, (X^TX)^-1,

are stochastically independent. It means that knowledge of a) does not give any knowledge of the value of b). In order to know the quality of predictions, b) must be computed in order to find out how well the model is performing.

          The situation is similar the one, where from the data values (x₁, ..., x_N) we want to estimate the mean and standard deviation of the normal distribution. If only the mean is estimated, there is no information on the value of the standard deviation.

The term x₀^T(X^TX)^-1x₀ can be viewed as having a distribution that is approximately proportional to a c² distribution (E(c²)=f and Var(c²)=2f). Typically, the H-method reduces this term by 50% or more, and obtains the same value for a) as standard methods that base results on traditional significance testing.

The reason is that the standard methods are only concerned with improving a). Thus they provide with no information on the size of b). When working with industrial data, the data are often of reduced statistical rank, which means that the relevant data of X are located in a subspace of the column space of X. By neglecting the part b) of the prediction variance, the standard methods typically provide with results that are often difficult or impossible to use by the company due to the bad prediction ability of the modelling results.

It is often difficult to identify the model that should be used to model the industrial data. Furthermore, even if the sciences are advanced they have not been developed with respect to tell about the future developments. The tradition that has established itself is not sufficient to provide with satisfactory solutions.

Back