g02ef:: Correlation and Regression Analysis (NAG Toolbox)

The general multiple linear regression model is defined by

y = β_{0} + X β + ε,

where

$y$ is a vector of $n$ observations on the dependent variable,
$β_{0}$ is an intercept coefficient,
$X$ is an $n$ by $p$ matrix of $p$ explanatory variables,
$β$ is a vector of $p$ unknown coefficients, and
$ε$ is a vector of length $n$ of unknown, Normally distributed, random errors.

nag_correg_linregm_fit_stepwise (g02ef) employs a full stepwise regression to select a subset of explanatory variables from the

p

available variables (the intercept is included in the model) and computes regression coefficients and their standard errors, and various other statistical quantities, by minimizing the sum of squares of residuals. The method applies repeatedly a forward selection step followed by a backward elimination step and halts when neither step updates the current model.

The criterion used to update a current model is the variance ratio of residual sum of squares. Let

s_{1}

and

s_{2}

be the residual sum of squares of the current model and this model after undergoing a single update, with degrees of freedom

q_{1}

and

q_{2}

, respectively. Then the condition:

\frac{(s_{2} - s_{1}) / (q_{2} - q_{1})}{s_{1} / q_{1}} > f_{1},

must be satisfied if a variable

k

will be considered for entry to the current model, and the condition:

\frac{(s_{1} - s_{2}) / (q_{1} - q_{2})}{s_{1} / q_{1}} < f_{2},

must be satisfied if a variable

k

will be considered for removal from the current model, where

f_{1}

and

f_{2}

are user-supplied values and

f_{2} \leq f_{1}

Explanatory variables are entered into and removed from the current model by using sweep operations on the correlation matrix

R

Z

, given by:

R = (\begin{matrix} 1 & \dots & r_{1 p} & r_{1 y} \\ ⋮ & ⋱ & ⋮ & ⋮ \\ r_{p 1} & \dots & 1 & r_{p y} \\ r_{y 1} & \dots & r_{y p} & 1 \end{matrix}),

where

r_{i j}

is the correlation between the explanatory variables

i

and

j

, for

i = 1, 2, \dots, p

and

j = 1, 2, \dots, p

, and

r_{y i}

(and

r_{i y}

) is the correlation between the response variable

y

and the

i

th explanatory variable, for

i = 1, 2, \dots, p

A sweep operation on the

k

th row and column (

k \leq p

) of

R

replaces:

\begin{array}{l} r_{k k} ​ by ​ - 1 / r_{k k}; \\ r_{i k} ​ by ​ r_{i k} / |r_{k k}|, i = 1, 2, \dots, p + 1 ​ ​ (i \neq k); \\ r_{k j} ​ by ​ r_{k j} / |r_{k k}|, j = 1, 2, \dots, p + 1 ​ ​ (j \neq k); \\ r_{i j} ​ by ​ r_{i j} - r_{i k} r_{k j} / |r_{k k}|,  ​ i = 1, 2, \dots, p + 1 ​ ​ (i \neq k); ​ j = 1, 2, \dots, p + 1 ​ ​ (j \neq k) . \end{array}

The

k

th explanatory variable is eligible for entry into the current model if it satisfies the collinearity tests:

r_{k k} > τ

and

(r_{i i} - \frac{r_{i k} r_{k i}}{r_{k k}}) τ \leq 1,

for a user-supplied value (

> 0

) of

τ

and where the index

i

runs over explanatory variables in the current model. The sweep operation is its own inverse, therefore pivoting on an explanatory variable

k

in the current model has the effect of removing it from the model.

Once the stepwise model selection procedure is finished, the function calculates:

(a)	the least squares estimate for the $i$ th explanatory variable included in the fitted model;
(b)	standard error estimates for each coefficient in the final model;
(c)	the square root of the mean square of residuals and its degrees of freedom;
(d)	the multiple correlation coefficient.

References

Parameters

Compulsory Input Parameters

Optional Input Parameters

Output Parameters

Error Indicators and Warnings

Cases prefixed with W are classified as warnings and do not generate an error of type NAG:error_n. See nag_issue_warnings.

Accuracy

Further Comments

Although the condition for removing or adding a variable to the current model is based on a ratio of variances, these values should not be interpreted as

F

-statistics with the usual interpretation of significance unless the probability levels are adjusted to account for correlations between variables under consideration and the number of possible updates (see, e.g., Draper and Smith (1985)).

Example

This example calculates a full stepwise model selection for the Hald data described in Dempster (1969). Means, the upper-triangular variance-covariance matrix and the sum of weights are calculated by nag_correg_ssqmat (g02bu). The NAG defined default monitor function nag_correg_linregm_fit_stepwise_sample_monfun (g02efh) is used to print information at each step of the model selection process.

function g02ef_example


fprintf('g02ef example results\n\n');

z   = [  7  26   6  60   78.5;
         1  29  15  52   74.3;
        11  56   8  20  104.3;
        11  31   8  47   87.6;
         7  52   6  33   95.9;
        11  55   9  22  109.2;
         3  71  17   6  102.7;
         1  31  22  44   72.5;
         2  54  18  22   93.1;
        21  47   4  26  115.9;
         1  40  23  34   83.8;
        11  66   9  12  113.3;
        10  68   8  12  109.4];
[n,m1] = size(z);
m = m1 - 1;

isx = ones(m,1,'int64');

% Compute sums of squares and cross-products of deviations from mean for z
[sw, wmean, c, ifail] = g02bu(z);

% Perform stepwise selection of variables
fout = 2;
[isx, b, se, rsq, rms, df, user, ifail] = ...
g02ef( ...
       int64(n), wmean, c, sw, isx, 'fout', fout, 'monfun', @monfun);

% Display results
fprintf('\nFitted Model Summary\n');
fprintf('Term              Estimate   Standard Error\n');
fprintf('Intercept:    %12.3e     %12.3e\n', b(1), se(1));
for j = 1:m
  if isx(j)==1 || isx(j)==2
    fprintf('Variable: %3d %12.3e     %12.3e\n', j, b(j+1), se(j+1));
  end
end
fprintf('\nRMS: %12.3e\n', rms);



function [user] = monfun(flag, var, val, user)

  switch flag
    case 'C'
      fprintf('\nVariable %d aliased\n', var);
    case 'S'
      fprintf('\nStarting Stepwise Selection\n');
    case 'F'
      fprintf('\nForward Selection\n');
    case 'V'
      fprintf('Variable %d  Variance ratio = %12.3f\n', var, val);
    case 'A'
      fprintf('\nAdding variable %d to model\n', var);
    case 'B'
      fprintf('\nBackward Selection\n');
    case 'D'
      fprintf('\nDropping variable %d from model\n', var);
    case 'K'
      fprintf('\nKeeping all current variables\n');
    case 'X'
      fprintf('\nFinished Stepwise Selection\n');
  end;

g02ef example results


Starting Stepwise Selection

Forward Selection
Variable 1  Variance ratio =       12.603
Variable 2  Variance ratio =       21.961
Variable 3  Variance ratio =        4.403
Variable 4  Variance ratio =       22.799

Adding variable 4 to model

Backward Selection
Variable 4  Variance ratio =       22.799

Keeping all current variables

Forward Selection
Variable 1  Variance ratio =      108.224
Variable 2  Variance ratio =        0.172
Variable 3  Variance ratio =       40.295

Adding variable 1 to model

Backward Selection
Variable 1  Variance ratio =      108.224
Variable 4  Variance ratio =      159.295

Keeping all current variables

Forward Selection
Variable 2  Variance ratio =        5.026
Variable 3  Variance ratio =        4.236

Adding variable 2 to model

Backward Selection
Variable 1  Variance ratio =      154.008
Variable 2  Variance ratio =        5.026
Variable 4  Variance ratio =        1.863

Dropping variable 4 from model

Forward Selection
Variable 3  Variance ratio =        1.832
Variable 4  Variance ratio =        1.863

Finished Stepwise Selection

Fitted Model Summary
Term              Estimate   Standard Error
Intercept:       5.258e+01        2.294e+00
Variable:   1    1.468e+00        1.213e-01
Variable:   2    6.623e-01        4.585e-02

RMS:    5.790e+00

NAG Toolbox: nag_correg_linregm_fit_stepwise (g02ef)

▸▿ Contents

Purpose

Syntax

Description