naginterfaces.library.correg.ridge_​opt

naginterfaces.library.correg.ridge_opt(x, isx, y, h, opt, niter, tol, orig, optloo, tau=0.0)

ridge_opt calculates a ridge regression, optimizing the ridge parameter according to one of four prediction error criteria.

For full information please refer to the NAG Library document for g02ka

https://support.nag.com/numeric/nl/nagdoc_30.2/flhtml/g02/g02kaf.html

Parameters

xfloat, array-like, shape $(n,m)$

The values of independent variables in the data matrix $X$.

isxint, array-like, shape $\left(m\right)$

Indicates which $m$ independent variables are included in the model.

$isx[j-1]=1$

The $j$th variable in $x$ will be included in the model.

$isx[j-1]=0$

Variable $j$ is excluded.

yfloat, array-like, shape $\left(n\right)$

The $n$ values of the dependent variable $y$.

hfloat

An initial value for the ridge regression parameter $h$; used as a starting point for the optimization.

optint

The measure of prediction error used to optimize the ridge regression parameter $h$. The value of $opt$ must be set equal to one of:

$opt=1$

Generalized cross-validation (GCV);

$opt=2$

Unbiased estimate of variance (UEV)

$opt=3$

Future prediction error (FPE)

$opt=4$

Bayesian information criteron (BIC).

niterint

The maximum number of iterations allowed to optimize the ridge regression parameter $h$.

tolfloat

Iterations of the ridge regression parameter $h$ will halt when consecutive values of $h$ lie within $tol$.

origint

If $orig=1$, the parameter estimates $b$ are calculated for the original data; otherwise $orig=2$ and the parameter estimates $\tilde{b}$ are calculated for the standardized data.

optlooint

If $optloo=2$, the leave-one-out cross-validation estimate of prediction error is calculated; otherwise no such estimate is calculated and $optloo=1$.

taufloat, optional

Singular values less than $tau$ of the SVD of the data matrix $X$ will be set equal to zero.

Returns

hfloat

$h$ is the optimized value of the ridge regression parameter $h$.

niterint

The number of iterations used to optimize the ridge regression parameter $h$ within $tol$.

nepfloat

The number of effective parameters, $\gamma$, in the model.

bfloat, ndarray, shape $(\text{ip}+1)$

Contains the intercept and parameter estimates for the fitted ridge regression model in the order indicated by $isx$. The first element of $b$ contains the estimate for the intercept; $b\left[\text{j}\right]$ contains the parameter estimate for the $\text{j}$th independent variable in the model, for $\text{j}=1,2,\dots ,\text{ip}$.

viffloat, ndarray, shape $\left(\text{ip}\right)$

The variance inflation factors in the order indicated by $isx$. For the $\text{j}$th independent variable in the model, $vif[\text{j}-1]$ is the value of $v_{\text{j}}$, for $\text{j}=1,2,\dots ,\text{ip}$.

resfloat, ndarray, shape $\left(n\right)$

$res[\text{i}-1]$ is the value of the $\text{i}$th residual for the fitted ridge regression model, for $\text{i}=1,2,\dots ,n$.

rssfloat

The sum of squares of residual values.

dfint

The degrees of freedom for the residual sum of squares $rss$.

perrfloat, ndarray, shape $\left(5\right)$

The first four elements contain, in this order, the measures of prediction error: GCV, UEV, FPE and BIC.

If $optloo=2$, $perr\left[4\right]$ is the LOOCV estimate of prediction error; otherwise $perr\left[4\right]$ is not referenced.

Raises

NagValueError

(errno $1$)

On entry, $n=⟨value⟩$.

Constraint: $n>1$.

(errno $1$)

On entry, $tau=⟨value⟩$.

Constraint: $tau\ge 0.0$.

(errno $1$)

On entry, $opt=⟨value⟩$.

Constraint: $opt=1$, $2$, $3$ or $4$.

(errno $1$)

On entry, $h=⟨value⟩$.

Constraint: $h>0.0$.

(errno $1$)

On entry, $optloo=⟨value⟩$.

Constraint: $optloo=1$ or $2$.

(errno $1$)

On entry, $tol=⟨value⟩$.

Constraint: $tol>0.0$.

(errno $1$)

On entry, $niter=⟨value⟩$.

Constraint: $niter\ge 1$.

(errno $1$)

On entry, $orig=⟨value⟩$.

Constraint: $orig=1$ or $2$.

(errno $2$)

On entry, $m=⟨value⟩$ and $n=⟨value⟩$.

Constraint: $m\le n$.

(errno $2$)

On entry, $\text{ip}=⟨value⟩$; $m=⟨value⟩$.

Constraint: $1\le \text{ip}\le m$.

(errno $2$)

On entry, $isx[⟨value⟩]=⟨value⟩$.

Constraint: $isx[j-1]=0$ or $1$.

(errno $2$)

On entry, $\text{ip}=⟨value⟩$.

Constraint: $sum\left(isx\right)=\text{ip}$.

(errno $3$)

SVD failed to converge.

Warns

NagAlgorithmicWarning

(errno $-1$)

Maximum number of iterations used.

Notes

A linear model has the form:

$$y=c+X\beta +ϵ\text{,}$$

where

$y$ is an $n\times 1$ matrix of values of a dependent variable;

$c$ is a scalar intercept term;

$X$ is an $n\times m$ matrix of values of independent variables;

$\beta$ is an $m\times 1$ matrix of unknown values of parameters;

$ϵ$ is an $n\times 1$ matrix of unknown random errors such that variance of $ϵ={\sigma }^{2}I$.

Let $\tilde{X}$ be the mean-centred $X$ and $\tilde{y}$ the mean-centred $y$. Furthermore, $\tilde{X}$ is scaled such that the diagonal elements of the cross product matrix ${\tilde{X}}^T\tilde{X}$ are one. The linear model now takes the form:

$$\tilde{y}=\tilde{X}\tilde{\beta }+ϵ\text{.}$$

Ridge regression estimates the parameters $\tilde{\beta }$ in a penalised least squares sense by finding the $\tilde{b}$ that minimizes

$${\Vert \tilde{X}\tilde{b}-\tilde{y}\Vert }^2+h{\Vert \tilde{b}\Vert }^2\text{,}h>0\text{,}$$

where $\parallel \cdot \parallel$ denotes the ${\ell }_2$-norm and $h$ is a scalar regularization or ridge parameter. For a given value of $h$, the parameter estimates $\tilde{b}$ are found by evaluating

$$\tilde{b}={({\tilde{X}}^T\tilde{X}+hI)}^{-1}{\tilde{X}}^T\tilde{y}\text{.}$$

Note that if $h=0$ the ridge regression solution is equivalent to the ordinary least squares solution.

Rather than calculate the inverse of (${\tilde{X}}^T\tilde{X}+hI$) directly, ridge_opt uses the singular value decomposition (SVD) of $\tilde{X}$. After decomposing $\tilde{X}$ into $UD{V}^T$ where $U$ and $V$ are orthogonal matrices and $D$ is a diagonal matrix, the parameter estimates become

$$\tilde{b}=V{(D^{T}D+hI)}^{-1}D{U}^T\tilde{y}\text{.}$$

A consequence of introducing the ridge parameter is that the effective number of parameters, $\gamma$, in the model is given by the sum of diagonal elements of

$$D^{T}D{(D^{T}D+hI)}^{-1}\text{,}$$

see Moody (1992) for details.

Any multi-collinearity in the design matrix $X$ may be highlighted by calculating the variance inflation factors for the fitted model. The $j$th variance inflation factor, $v_j$, is a scaled version of the multiple correlation coefficient between independent variable $j$ and the other independent variables, $R_j$, and is given by

$$v_j=\frac{1}{1-R_j}\text{,}j=1,2,\dots ,m\text{.}$$

The $m$ variance inflation factors are calculated as the diagonal elements of the matrix:

$${({\tilde{X}}^T\tilde{X}+hI)}^{-1}{\tilde{X}}^T\tilde{X}{({\tilde{X}}^T\tilde{X}+hI)}^{-1}\text{,}$$

which, using the SVD of $\tilde{X}$, is equivalent to the diagonal elements of the matrix:

$$V{(D^{T}D+hI)}^{-1}{D}^{T}D{(D^{T}D+hI)}^{-1}{V}^T\text{.}$$

Although parameter estimates $\tilde{b}$ are calculated by using $\tilde{X}$, it is usual to report the parameter estimates $b$ associated with $X$. These are calculated from $\tilde{b}$, and the means and scalings of $X$. Optionally, either $\tilde{b}$ or $b$ may be calculated.

The method can adopt one of four criteria to minimize while calculating a suitable value for $h$:

1. Generalized cross-validation (GCV):

   $$\frac{ns}{{(n-\gamma )}^2}\text{;}$$

2. Unbiased estimate of variance (UEV):

   $$\frac{s}{n-\gamma }\text{;}$$

3. Future prediction error (FPE):

   $$\frac{1}{n}(s+\frac{2\gamma s}{n-\gamma })\text{;}$$

4. Bayesian information criterion (BIC):

   $$\frac{1}{n}(s+\frac{\log \left(n\right)\gamma s}{n-\gamma })\text{;}$$

where $s$ is the sum of squares of residuals. However, the function returns all four of the above prediction errors regardless of the one selected to minimize the ridge parameter, $h$. Furthermore, the function will optionally return the leave-one-out cross-validation (LOOCV) prediction error.

References

Hastie, T, Tibshirani, R and Friedman, J, 2003, The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer Series in Statistics

Moody, J.E., 1992, The effective number of parameters: An analysis of generalisation and regularisation in nonlinear learning systems, In: Neural Information Processing Systems, (eds J E Moody, S J Hanson, and R P Lippmann), 4, 847–854, Morgan Kaufmann San Mateo CA

See also

naginterfaces.library.examples.correg.ridge_opt_ex.main()