naginterfaces.library.correg.ridge

naginterfaces.library.correg.ridge(x, isx, y, h, wantb, wantvf, pec=None)

ridge calculates a ridge regression, with ridge parameters supplied by you.

For full information please refer to the NAG Library document for g02kb

https://support.nag.com/numeric/nl/nagdoc_30/flhtml/g02/g02kbf.html

Parameters

xfloat, array-like, shape $(n,m)$

The values of independent variables in the data matrix $X$.

isxint, array-like, shape $\left(m\right)$

Indicates which $m$ independent variables are included in the model.

$isx[j-1]=1$

The $j$th variable in $x$ will be included in the model.

$isx[j-1]=0$

Variable $j$ is excluded.

yfloat, array-like, shape $\left(n\right)$

The $n$ values of the dependent variable $y$.

hfloat, array-like, shape $\left(\text{lh}\right)$

$h[j-1]$ is the value of the $j$th ridge parameter $h$.

wantbint

Defines the options for parameter estimates.

$wantb=0$

Parameter estimates are not calculated and $b$ is not referenced.

$wantb=1$

Parameter estimates $b$ are calculated for the original data.

$wantb=2$

Parameter estimates $\tilde{b}$ are calculated for the standardized data.

wantvfint

Defines the options for variance inflation factors.

$wantvf=0$

Variance inflation factors are not calculated and the array $vf$ is not referenced.

$wantvf=1$

Variance inflation factors are calculated.

pecNone or str, length 1, array-like, shape $\left(\text{lpec}\right)$, optional

If $pec$ is not None, $pec[\text{j}-1]$ defines the $\text{j}$th prediction error, for $\text{j}=1,2,\dots ,\text{lpec}$; otherwise $pec$ is not referenced.

$pec[j-1]=\text{'B'}$

Bayesian information criterion (BIC).

$pec[j-1]=\text{'F'}$

Future prediction error (FPE).

$pec[j-1]=\text{'G'}$

Generalized cross-validation (GCV).

$pec[j-1]=\text{'L'}$

Leave-one-out cross-validation (LOOCV).

$pec[j-1]=\text{'U'}$

Unbiased estimate of variance (UEV).

Returns

nepfloat, ndarray, shape $\left(\text{lh}\right)$

$nep[\text{j}-1]$ is the number of effective parameters, $\gamma$, in the $\text{j}$th model, for $\text{j}=1,2,\dots ,\text{lh}$.

bfloat, ndarray, shape $(:,:)$

If $wantb\ne 0$, $b$ contains the intercept and parameter estimates for the fitted ridge regression model in the order indicated by $isx$. $b[0,\text{j}-1]$, for $\text{j}=1,2,\dots ,\text{lh}$, contains the estimate for the intercept; $b[\text{i},j-1]$ contains the parameter estimate for the $\text{i}$th independent variable in the model fitted with ridge parameter $h[j-1]$, for $\text{i}=1,2,\dots ,\text{ip}$.

vffloat, ndarray, shape $(:,:)$

If $wantvf=1$, the variance inflation factors. For the $\text{i}$th independent variable in a model fitted with ridge parameter $h[j-1]$, $vf[\text{i}-1,j-1]$ is the value of $v_{\text{i}}$, for $\text{i}=1,2,\dots ,\text{ip}$.

peNone or float, ndarray, shape $(:,:)$

If $pec$ is None on entry, $pe$ is None; otherwise $pe[\text{i}-1,\text{j}-1]$ contains the prediction error of criterion $pec[\text{i}-1]$ for the model fitted with ridge parameter $h[\text{j}-1]$, for $\text{j}=1,2,\dots ,\text{lh}$, for $\text{i}=1,2,\dots ,\text{lpec}$.

Raises

NagValueError

(errno $1$)

On entry, $n=⟨value⟩$.

Constraint: $n\ge 1$.

(errno $1$)

On entry, $m=⟨value⟩$ and $n=⟨value⟩$.

Constraint: $m\le n$.

(errno $1$)

On entry, $h[j-1]<0$ for at least one $j$.

Constraint: $h[j-1]\le 0.0$, for all $j$.

(errno $1$)

On entry, $\text{lh}=⟨value⟩$.

Constraint: $\text{lh}>0$.

(errno $1$)

On entry, $wantb=⟨value⟩$.

Constraint: $wantb=0$, $1$ or $2$.

(errno $1$)

On entry, $wantvf=⟨value⟩$.

Constraint: $wantvf=0$ or $1$.

(errno $1$)

On entry, $pec[j-1]$ is invalid for at least one $j$.

Constraint: if $pec$ is not None, $pec[\text{j}-1]=\text{'B'}$, $\text{'F'}$, $\text{'G'}$, $\text{'L'}$ or $\text{'U'}$, for all $j$.

(errno $2$)

On entry, $isx[j-1]\ne 0$ or $1$ for at least one $j$.

Constraint: $isx[j-1]=0$ or $1$, for all $j$.

(errno $2$)

On entry, $\text{ip}$ is not equal to the sum of elements in $isx$.

Constraint: exactly $\text{ip}$ elements of $isx$ must be equal to $1$.

(errno $2$)

On entry, $\text{ldb}=⟨value⟩$ and $\text{ip}=⟨value⟩$.

Constraint: if $wantb\ne 0$, $\text{ldb}\ge \text{ip}+1$.

(errno $2$)

On entry, $\text{ldvf}=⟨value⟩$ and $\text{ip}=⟨value⟩$.

Constraint: if $wantvf\ne 0$, $\text{ldvf}\ge \text{ip}$.

(errno $3$)

On entry, $wantb=0$ and $wantvf=0$.

Constraint: $wantb=0$, $wantvf=1$.

Notes

A linear model has the form:

$$y=c+X\beta +ϵ\text{,}$$

where

$y$ is an $n\times 1$ matrix of values of a dependent variable;

$c$ is a scalar intercept term;

$X$ is an $n\times m$ matrix of values of independent variables;

$\beta$ is an $m\times 1$ matrix of unknown values of parameters;

$ϵ$ is an $n\times 1$ matrix of unknown random errors such that variance of ${ϵ=\sigma }^{2}I$.

Let $\tilde{X}$ be the mean-centred $X$ and $\tilde{y}$ the mean-centred $y$. Furthermore, $\tilde{X}$ is scaled such that the diagonal elements of the cross product matrix ${\tilde{X}}^T\tilde{X}$ are one. The linear model now takes the form:

$$\tilde{y}=\tilde{X}\tilde{\beta }+ϵ\text{.}$$

Ridge regression estimates the parameters $\tilde{\beta }$ in a penalised least squares sense by finding the $\tilde{b}$ that minimizes

$${\Vert \tilde{X}\tilde{b}-\tilde{y}\Vert }^2+h{\Vert \tilde{b}\Vert }^2\text{,}h>0\text{,}$$

where $\parallel \cdot \parallel$ denotes the ${\ell }_2$-norm and $h$ is a scalar regularization or ridge parameter. For a given value of $h$, the parameters estimates $\tilde{b}$ are found by evaluating

$$\tilde{b}={({\tilde{X}}^T\tilde{X}+hI)}^{-1}{\tilde{X}}^T\tilde{y}\text{.}$$

Note that if $h=0$ the ridge regression solution is equivalent to the ordinary least squares solution.

Rather than calculate the inverse of (${\tilde{X}}^T\tilde{X}+hI$) directly, ridge uses the singular value decomposition (SVD) of $\tilde{X}$. After decomposing $\tilde{X}$ into $UD{V}^T$ where $U$ and $V$ are orthogonal matrices and $D$ is a diagonal matrix, the parameter estimates become

$$\tilde{b}=V{(D^{T}D+hI)}^{-1}D{U}^T\tilde{y}\text{.}$$

A consequence of introducing the ridge parameter is that the effective number of parameters, $\gamma$, in the model is given by the sum of diagonal elements of

$$D^{T}D{(D^{T}D+hI)}^{-1}\text{,}$$

see Moody (1992) for details.

Any multi-collinearity in the design matrix $X$ may be highlighted by calculating the variance inflation factors for the fitted model. The $j$th variance inflation factor, $v_j$, is a scaled version of the multiple correlation coefficient between independent variable $j$ and the other independent variables, $R_j$, and is given by

$$v_j=\frac{1}{1-R_j}\text{,}j=1,2,\dots ,m\text{.}$$

The $m$ variance inflation factors are calculated as the diagonal elements of the matrix:

$${({\tilde{X}}^T\tilde{X}+hI)}^{-1}{\tilde{X}}^T\tilde{X}{({\tilde{X}}^T\tilde{X}+hI)}^{-1}\text{,}$$

which, using the SVD of $\tilde{X}$, is equivalent to the diagonal elements of the matrix:

$$V{(D^{T}D+hI)}^{-1}{D}^{T}D{(D^{T}D+hI)}^{-1}{V}^T\text{.}$$

Given a value of $h$, any or all of the following prediction criteria are available:

1. Generalized cross-validation (GCV):

   $$\frac{ns}{{(n-\gamma )}^2}\text{;}$$

2. Unbiased estimate of variance (UEV):

   $$\frac{s}{n-\gamma }\text{;}$$

3. Future prediction error (FPE):

   $$\frac{1}{n}(s+\frac{2\gamma s}{n-\gamma })\text{;}$$

4. Bayesian information criterion (BIC):

   $$\frac{1}{n}(s+\frac{\log \left(n\right)\gamma s}{n-\gamma })\text{;}$$

5. Leave-one-out cross-validation (LOOCV),

where $s$ is the sum of squares of residuals.

Although parameter estimates $\tilde{b}$ are calculated by using $\tilde{X}$, it is usual to report the parameter estimates $b$ associated with $X$. These are calculated from $\tilde{b}$, and the means and scalings of $X$. Optionally, either $\tilde{b}$ or $b$ may be calculated.

References

Hastie, T, Tibshirani, R and Friedman, J, 2003, The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer Series in Statistics

Moody, J.E., 1992, The effective number of parameters: An analysis of generalisation and regularisation in nonlinear learning systems, In: Neural Information Processing Systems, (eds J E Moody, S J Hanson, and R P Lippmann), 4, 847–854, Morgan Kaufmann San Mateo CA