naginterfaces.library.opt.handle_​solve_​nldf

naginterfaces.library.opt.handle_solve_nldf(handle, lsqfun, lsqgrd, x, nres, confun=None, congrd=None, monit=None, data=None, io_manager=None)

handle_solve_nldf is a solver from the NAG optimization modelling suite for general nonlinear data-fitting problems with constraints. Various loss and regularization functions are supported.

Note: this function uses optional algorithmic parameters, see also: handle_opt_set(), handle_opt_get().

For full information please refer to the NAG Library document for e04gn

https://support.nag.com/numeric/nl/nagdoc_30.2/flhtml/e04/e04gnf.html

Parameters

handleHandle

The handle to the problem. It needs to be initialized (e.g., by handle_init()) and to hold a problem formulation compatible with handle_solve_nldf. It must not be changed between calls to the NAG optimization modelling suite.

lsqfuncallable (rx, inform) = lsqfun(x, nres, inform, data=None)

$lsqfun$ must evaluate the value of the nonlinear residuals, $r_i\left(x\right):=y_i-\varphi (t_i;x),i=1,\dots ,n_{res}$, at a specified point $x$.

Parameters

xfloat, ndarray, shape $\left(\text{nvar}\right)$

$x$, the vector of variable values at which the residuals, $r_i$, are to be evaluated.

nresint

$n_{\text{res}}$, the current number of residuals in the model.

informint

A non-negative value.

dataarbitrary, optional, modifiable in place

User-communication data for callback functions.

Returns

rxfloat, array-like, shape $\left(nres\right)$

The value of the residual vector, $r\left(x\right)$, evaluated at $x$.

informint

May be used to indicate that some residuals could not be computed at the requested point. This can be done by setting $inform$ to a negative value. The solver will attempt a rescue procedure and request an alternative point. If the rescue procedure fails, the solver will exit with $errno$ = 25.

lsqgrdcallable inform = lsqgrd(x, nres, rdx, inform, data=None)

$lsqgrd$ evaluates the residual gradients, $\nabla r_i\left(x\right)$, at a specified point $x$.

Parameters

xfloat, ndarray, shape $\left(\text{nvar}\right)$

$x$, the vector of variable values at which the residual gradients, $\nabla r_i\left(x\right)$, are to be evaluated.

nresint

$n_{\text{res}}$, the current number of residuals in the model.

rdxfloat, ndarray, shape $\left(\text{nnzrd}\right)$, to be modified in place

On entry: the elements should only be assigned and not referenced.

On exit: the vector containing the nonzero residual gradients evaluated at $x$,

$$\nabla r\left(x\right)=[\nabla r_1\left(x\right),\nabla r_2\left(x\right),\dots ,\nabla r_{n_{\text{res}}}\left(x\right)]\text{,}$$

where

$$\nabla r_i\left(x\right)={[\frac{\partial r_i\left(x\right)}{\partial x_1},\dots ,\frac{\partial r_i\left(x\right)}{\partial x_{n_{\text{var}}}}]}^T\text{.}$$

The elements must be stored in the same order as the defined sparsity pattern provided in the call to handle_set_nlnls().

informint

A non-negative value.

dataarbitrary, optional, modifiable in place

User-communication data for callback functions.

Returns

informint

May be used to indicate that the residual gradients could not be computed at the requested point. This can be done by setting $inform$ to a negative value. The solver will attempt a rescue procedure and request an alternative point. If the rescue procedure fails, the solver will exit with $errno$ = 25.

xfloat, array-like, shape $\left(\text{nvar}\right)$

$x_0$, the initial estimates of the variables, $x$.

nresint

$n_{\text{res}}$, the current number of residuals in the model.

confunNone or callable (gx, inform) = confun(x, ncnln, inform, data=None), optional

Note: if this argument is None then a NAG-supplied facility will be used.

$confun$ must calculate the values of the $m_g$-element vector $g_i\left(x\right)$ of nonlinear constraint functions at a specified value of the $n_{\text{var}}$-element vector of $x$ variables.

If there are no nonlinear constraints then $confun$ will never be called by handle_solve_nldf and it may be None.

Parameters

xfloat, ndarray, shape $\left(\text{nvar}\right)$

The vector $x$ of variable values at which the constraint functions are to be evaluated.

ncnlnint

$m_g$, the number of nonlinear constraints, as specified in an earlier call to handle_set_nlnconstr().

informint

A non-negative value.

dataarbitrary, optional, modifiable in place

User-communication data for callback functions.

Returns

gxfloat, array-like, shape $\left(ncnln\right)$

The $m_g$ values of the nonlinear constraint functions at $x$.

informint

Must be set to a value describing the action to be taken by the solver on return from $confun$. Specifically, if the value is negative, then the value of $gx$ will be discarded and the solver will either attempt to find a different trial point or terminate immediately with $errno$ = 25; otherwise, the solver will proceed normally.

congrdNone or callable inform = congrd(x, gdx, inform, data=None), optional

Note: if this argument is None then a NAG-supplied facility will be used.

$congrd$ must calculate the nonzero values of the sparse Jacobian of the nonlinear constraint functions, $\frac{\partial g_i}{\partial x}$, at a specified value of the $n_{\text{var}}$-element vector of $x$ variables.

If there are no nonlinear constraints, $congrd$ will never be called by handle_solve_nldf and $congrd$ may be None.

Parameters

xfloat, ndarray, shape $\left(\text{nvar}\right)$

The vector $x$ of variable values at which the Jacobian of the constraint functions is to be evaluated.

gdxfloat, ndarray, shape $\left(\text{nnzgd}\right)$, to be modified in place

On entry: the elements should only be assigned and not referenced.

On exit: the nonzero values of the Jacobian of the nonlinear constraints, in the order specified by $\text{irowgd}$ and $\text{icolgd}$ in an earlier call to handle_set_nlnconstr(). $gdx[\text{i}-1]$ will be the gradient $\frac{\partial g_j}{\partial x_k}$, where $j=\text{irowgd}[\text{i}-1]$ and $k=\text{icolgd}[\text{i}-1]$.

informint

A non-negative value.

dataarbitrary, optional, modifiable in place

User-communication data for callback functions.

Returns

informint

Must be set to a value describing the action to be taken by the solver on return from $congrd$. Specifically, if the value is negative the solution of the current problem will terminate immediately with $errno$ = 25; otherwise, computations will continue.

monitNone or callable monit(x, rinfo, stats, data=None), optional

Note: if this argument is None then a NAG-supplied facility will be used.

$monit$ is provided to enable monitoring of the progress of the optimization and, if necessary, to halt the optimization process.

If no monitoring is required, $monit$ may be None.

$monit$ is called at the end of every $i$th step where $i$ is controlled by the option ‘NLDF Monitor Frequency’ (if the value is $0$, $monit$ is not called).

Parameters

xfloat, ndarray, shape $\left(\text{nvar}\right)$

The current best point.

rinfofloat, ndarray, shape $\left(100\right)$

Best objective value computed and various indicators (the values are as described in the main argument $rinfo$).

statsfloat, ndarray, shape $\left(100\right)$

Solver statistics at monitoring steps or at the end of the current iteration (the values are as described in the main argument $stats$).

dataarbitrary, optional, modifiable in place

User-communication data for callback functions.

dataarbitrary, optional

User-communication data for callback functions.

io_managerFileObjManager, optional

Manager for I/O in this routine.

Returns

xfloat, ndarray, shape $\left(\text{nvar}\right)$

The final values of the variables, $x$.

rxfloat, ndarray, shape $\left(nres\right)$

The values of the residuals at the final point given in $x$.

rinfofloat, ndarray, shape $\left(100\right)$

Objective value and various indicators at monitoring steps or at the end of the final iteration. The measures are:

$0$	Objective function value, $f\left(x\right)$ in [equation].
$1$	Loss function value, ${\sum }_{i=1}^{n_{\text{res}}}\chi \left(r_i\left(x\right)\right)$ in [equation].
$2$	Regularization term value, ${\sum }_{i=1}^{n_{\text{var}}}\psi \left(x_i\right)$ in [equation].
$3$	Solution optimality measure.
$4-99$	Reserved for future use.

statsfloat, ndarray, shape $\left(100\right)$

Solver statistics at monitoring steps or at the end of the final iteration:

$0$	Number of iterations performed.
$1$	Total time in seconds spent in the solver. It includes time spent in user-supplied subroutines.
$2-99$	Reserved for future use.

Other Parameters

‘Defaults’valueless

This special keyword may be used to reset all options to their default values. Any value given with this keyword will be ignored.

‘NLDF Iteration Limit’int

Default $=10000000$

The maximum number of iterations to be performed by handle_solve_nldf. If this limit is reached, then the solver will terminate with $errno$ = 22.

Constraint: $\text{‘NLDF Iteration Limit'}\ge 1$.

‘NLDF Monitor Frequency’int

Default $=0$

If $\text{‘NLDF Monitor Frequency'}>0$, the user-supplied function $monit$ will be called at the end of every $i$th step for monitoring purposes.

Constraint: $\text{‘NLDF Monitor Frequency'}\ge 0$.

‘NLDF Stop Tolerance’float

Default $=max({10}^{-6},\sqrt{ϵ})$

This argument sets the value of ${ϵ}_{\text{tol}}$ which specifies the tolerance for the optimality measure.

When both loss function and regularization are differentiable, and with only simple bound constraints, the optimality measures are defined by [equation] and [equation] and ‘NLDF Stop Tolerance’ is passed to the solver handle_solve_bounds_foas() as ‘FOAS Stop Tolerance’.

When any of the loss function or regularization is non-differentiable, or there presents linear, quadratic or general nonlinear constraint, the optimality measure is defined by (5) and ‘NLDF Stop Tolerance’ is passed to the solver handle_solve_ipopt() as ‘Stop Tolerance 1’.

Constraint: $0\le {ϵ}_{\text{tol}}<1$.

‘NLDF Loss Function Type’str

Default $=$’L2’

This argument sets the loss function type used in the objective function.

Constraint: $\text{‘NLDF Loss Function Type'}=\text{'HUBER'}$, $\text{'L2'}$, $\text{'CAUCHY'}$, $\text{'ATAN'}$, $\text{'SMOOTHL1'}$, $\text{'LINF'}$, $\text{'L1'}$ or $\text{'QUANTILE'}$.

‘Reg Term Type’str

Default $=$’OFF’

This argument sets the regularization function type used in the objective function.

Note: if there is no residual in the model, regularization will be turned off.

Constraint: $\text{‘Reg Term Type'}=\text{'OFF'}$, $\text{'L2'}$, $\text{'L1'}$ or $\text{'ELASTIC NET'}$.

‘NLDF Huber Function Width’float

Default $=1.0$

Sets the parameter $d_{\text{hub}}$ defined in the Huber loss function [equation].

Constraint: $\text{‘NLDF Huber Function Width'}>0$.

‘NLDF Cauchy Function Sharpness’float

Default $=1.0$

Sets the parameter $d_{\text{cau}}$ defined in the Cauchy loss function [equation].

Constraint: $\text{‘NLDF Cauchy Function Sharpness'}>0$.

‘NLDF SmoothL1 Function Width’float

Default $=1.0$

Sets the parameter $d_{\text{sl}1}$ defined in the SmoothL1 loss function [equation].

Constraint: $\text{‘NLDF SmoothL1 Function Width'}>0$.

‘NLDF Quantile Parameter’float

Default $=0.5$

Sets the parameter $d_{\text{qnt}}$ defined in the Quantile loss function [equation].

Constraint: $0<\text{‘NLDF Quantile Parameter'}<1$.

‘Reg Coefficient’float

Default $=1.0$

Sets the regularization coefficient $\rho$ in the definition of the objective function [equation]. Note: if set to $0$, regularization will be turned off.

Constraint: $\text{‘Reg Coefficient'}\ge 0$.

‘NLDF Elastic Net Alpha’float

Default $=0.5$

Sets the regularization coefficient $\alpha$ in the definition of the elastic net regularization function [equation].

Note: if set to $0$, the regularization function reduces to the $l_2$-norm. In this case, only the regularization coefficient ‘Reg Coefficient’ is used, and the value of ‘NLDF Elastic Net Beta’ is ignored.

Constraint: $\text{‘NLDF Elastic Net Alpha'}\ge 0$.

‘NLDF Elastic Net Beta’float

Default $=0.5$

Sets the regularization coefficient $\beta$ in the definition of the elastic net regularization function [equation].

Note: if set to $0$, the regularization function reduces to the $l_1$-norm. In this case, only the regularization coefficient ‘Reg Coefficient’ is used, and the value of ‘NLDF Elastic Net Alpha’ is ignored.

Constraint: $\text{‘NLDF Elastic Net Beta'}\ge 0$.

‘Infinite Bound Size’float

Default $={10}^{20}$

This defines the ‘infinite’ bound $\text{bigbnd}$ in the definition of the problem constraints. Any upper bound greater than or equal to $\text{bigbnd}$ will be regarded as $+\infty$ (and similarly any lower bound less than or equal to $-\text{bigbnd}$ will be regarded as $-\infty$). Note that a modification of this option does not influence constraints which have already been defined; only the constraints formulated after the change will be affected.

Constraint: $\text{‘Infinite Bound Size'}\ge 1000$.

‘Monitoring File’int

Default $=-1$

If $i\ge 0$, the unit number for the secondary (monitoring) output. If $\text{‘Monitoring File'}=-1$, no secondary output is provided. The information output to this unit is controlled by ‘Monitoring Level’.

Constraint: $\text{‘Monitoring File'}\ge -1$.

‘Monitoring Level’int

Default $=4$

This argument sets the amount of information detail that will be printed by the solver to the secondary output. The meaning of the levels is the same as for ‘Print Level’.

Constraint: $0\le \text{‘Monitoring Level'}\le 5$.

‘Print File’int

Default $=\text{advisory message unit number}$

If $i\ge 0$, the unit number for the primary output of the solver. If $\text{‘Print File'}=-1$, the primary output is completely turned off independently of other settings. The default value is the advisory message unit number at the time of the initialization of the options, e.g., at the initialization of the handle. The information output to this unit is controlled by ‘Print Level’.

Constraint: $\text{‘Print File'}\ge -1$.

‘Print Level’int

Default $=2$

This argument defines how detailed information should be printed by the solver to the primary and secondary output.

$i$	Output
$0$	No output from the solver.
$1$	The Header and Summary.
$2$, $3$, $4$, $5$	Additionally, the Iteration log.

Constraint: $0\le \text{‘Print Level'}\le 5$.

‘Print Options’str

Default $=\text{'YES'}$

If $\text{‘Print Options'}=\text{'YES'}$, a listing of options will be printed to the primary output and is always printed to the secondary output.

Constraint: $\text{‘Print Options'}=\text{'YES'}$ or $\text{'NO'}$.

‘Print Solution’str

Default $=\text{'NO'}$

If $\text{‘Print Solution'}=\text{'YES'}$ or $\text{'X'}$, the final values of the primal variables are printed on the primary and secondary outputs.

Constraint: $\text{‘Print Solution'}=\text{'YES'}$, $\text{'NO'}$ or $\text{'X'}$.

‘Stats Time’str

Default $=\text{'NO'}$

This argument turns on timing. This might be helpful for a choice of different solving approaches. It is possible to choose between CPU and wall clock time. Choice ‘YES’ is equivalent to ‘WALL CLOCK’.

Constraint: $\text{‘Stats Time'}=\text{'YES'}$, $\text{'NO'}$, $\text{'CPU'}$ or $\text{'WALL CLOCK'}$.

‘Time Limit’float

Default $={10}^6$

A limit to the number of seconds that the solver can use to solve one problem. If at the end of an iteration this limit is exceeded, the solver will terminate with $errno$ = 23.

Constraint: $\text{‘Time Limit'}>0$.

Raises

NagValueError

(errno $-199$)

handle_solve_nldf is not available in this implementation.

(errno $1$)

$handle$ has not been initialized.

(errno $1$)

$handle$ does not belong to the NAG optimization modelling suite, has not been initialized properly or is corrupted.

(errno $1$)

$handle$ has not been initialized properly or is corrupted.

(errno $2$)

This solver does not support the model defined in the handle.

(errno $2$)

The problem is already being solved.

(errno $4$)

On entry, $\text{nvar}=⟨value⟩$, expected $value=⟨value⟩$.

Constraint: $\text{nvar}$ must match the current number of variables of the model in the $handle$.

(errno $4$)

On entry, $nres=⟨value⟩$, expected $value=⟨value⟩$.

Constraint: $nres$ must match the current number of residuals defined in the $handle$.

(errno $7$)

Please provide a proper $confun$ function.

(errno $7$)

Please provide a proper $congrd$ function.

(errno $21$)

The current starting point is unusable.

(errno $28$)

The solver terminated with not enough degrees of freedom.

(errno $51$)

The solver detected an infeasible problem.

Warns

NagAlgorithmicWarning

(errno $50$)

Problem was solved to an acceptable level; full accuracy was not achieved.

NagAlgorithmicMajorWarning

(errno $22$)

Maximum number of iterations reached.

(errno $23$)

The solver terminated after the maximum time allowed was exhausted.

(errno $24$)

The solver was terminated because no further progress could be achieved.

(errno $25$)

Invalid number detected in user function.

(errno $28$)

The solver terminated after failure during line search.

(errno $28$)

The solver terminated after an error in the step computation.

NagCallbackTerminateWarning

(errno $20$)

User requested termination during a monitoring step.

Notes

handle_solve_nldf solves a data-fitting problem of the form

$$\begin{array}{c}\begin{array}{cc}{minimize}_{x\in R^{n_{\text{var}}}}& f\left(x\right)={\sum }_{i=1}^{n_{\text{res}}}\chi \left(r_i\left(x\right)\right)+\rho {\sum }_{i=1}^{n_{\text{var}}}\psi \left(x_i\right)\\ \text{subject to}& l_g\le g\left(x\right)\le u_g\text{,}\\ & \frac{1}{2}{x}^T{Q}_{i}x+p_i^{T}x+s_i\le 0\text{,}1\le i\le m_Q\text{,}\\ & l_B\le Bx\le u_B\text{,}\\ & l_x\le x\le u_x\text{,}\end{array}\end{array}$$

where

$n_{\text{var}}$ is the number of decision variables,

$m_g$ is the number of the nonlinear constraints and $g\left(x\right)$, $l_g$ and $u_g$ are $m_g$-dimensional vectors,

$m_Q$ is the number of quadratic constraints,

$m_B$ is the number of the linear constraints and $B$ is a $m_B\times n_{\text{var}}$ matrix, $l_B$ and $u_B$ are $m_B$-dimensional vectors,

there are $n_{\text{var}}$ box constraints and $l_x$ and $u_x$ are $n_{\text{var}}$-dimensional vectors.

Here,

$x$ is an $n_{\text{var}}$-dimensional vector representing the model parameters,

$\chi$ is the loss function,

$\psi$ is the regularization function,

$\rho$ is the regularization coefficient,

$n_{\text{res}}$ is the number of residuals and $r_i$ is the $i$th residual, which is defined as

$$r_i\left(x\right)=y_i-\varphi (t_i;x)\text{,}i=1,\dots ,n_{\text{res}}$$

where $\varphi (t_i;x)$ is the predicted value of the $i$th data point, given $x$. For the $i$th data point, $y_i$ and $t_i$ are the observed values of the independent and dependant variables respectively.

The available loss and regularization function types are summarized in Table [label omitted], where $d$ is the function parameter and $I\left(L\right)$ denotes an indicator function taking the value $1$ if the logical expression $L$ is true and $0$ otherwise. Loss function and regularization types can be specified by options ‘NLDF Loss Function Type’ and ‘Reg Term Type’, respectively. For example, set $\text{‘NLDF Loss Function Type'}=\text{'LINF'}$ and $\text{‘Reg Term Type'}=\text{'L2'}$ to use $l_{\infty }$-norm loss function with $l_2$-norm (Ridge) regularization. See Algorithmic Details for more details on the loss functions.

Loss function	$\chi \left(r_i\right)$	‘NLDF Loss Function Type’
$l_2$-norm	$r_i^2$	‘L2’
$l_1$-norm	$\left|r_i\right|$	‘L1’
$l_{\infty }$-norm	${max}_{1\le j\le n_{\text{res}}}\left|r_j\right|/n_{\text{res}}$	‘LINF’
Huber (see [equation])	$\{\begin{array}{cc}0.5\ast r_i^2& \text{if}\left|r_i\right|<d\\ d\ast (\left|r_i\right|-0.5\ast d)& \text{otherwise}\end{array}$	‘HUBER’
Cauchy (see [equation])	$ln(1+{\left(r_i/d\right)}^2)$	‘CAUCHY’
Atan	$\arctan \left(r_i^2\right)$	‘ATAN’
SmoothL1 (see [equation])	$\{\begin{array}{cc}0.5\ast r_i^2/d& \text{if}\left|r_i\right|<d\\ \left|r_i\right|-0.5\ast d& \text{otherwise}\end{array}$	‘SMOOTHL1’
Quantile (see [equation])	$r_i\ast (d-I_{(r_i<0)})$	‘QUANTILE’
Regularization	$\psi \left(x_i\right)$	‘Reg Term Type’
Lasso ($l_1$-norm)	$\left|x_i\right|$	‘L1’
Ridge ($l_2$-norm)	$x_i^2$	‘L2’
Elastic net	$\alpha \left|x_i\right|+\beta x_i^2$	‘ELASTIC NET’

handle_solve_nldf serves as a solver for problems stored as a $handle$. The $handle$ points to an internal data structure which defines the problem and serves as a means of communication for functions in the NAG optimization modelling suite. After the $handle$ has been initialized (e.g., handle_init() has been called), handle_set_nlnls() can be used to add a model and define its residual sparsity structure. handle_set_qconstr() and handle_set_qconstr_fac() may be used to set or modify quadratic constraints. Linear constraints $l_B$, $B$, $u_B$ are handled by handle_set_linconstr(). Variable box bounds $l_x$ and $u_x$ can be specified with handle_set_simplebounds(), and handle_set_nlnconstr() can set or modify nonlinear constraints. Once the problem is fully described, the $handle$ may be passed to the solver handle_solve_nldf. When the $handle$ is no longer needed, handle_free() should be called to destroy it and deallocate the memory held within. See the E04 Introduction for more details about the NAG optimization modelling suite.

Nonlinear Programming (NLP) solvers handle_solve_bounds_foas() and handle_solve_ipopt() are used as solver engines by handle_solve_nldf, which defines the selected loss function and regularization, then transforms the problem into standard form that the NLP solvers allow. For best performance, when the objective function $f\left(x\right)$ is differentiable and without any constraint other than simple bound constraints, handle_solve_bounds_foas() is used. For non-differentiable objective functions or cases where constraints other than simple variable bounds are present, handle_solve_ipopt() is used. See Algorithmic Details and handle_solve_ipopt() for more details on algorithmic details.

The algorithm behaviour can be modified by various options (see Other Parameters) which can be set by handle_opt_set() and handle_opt_set_file() anytime between the initialization of the $handle$ by e.g., handle_init() and a call to the solver. Once the solver has finished, options may be modified for the next solve. The solver may be called repeatedly with various starting points and/or options. Option getter handle_opt_get() can be called to retrieve the current value of any option.

See also

naginterfaces.library.examples.opt.handle_solve_nldf_ex.main()