NAG CL Interface
e04gnc (handle_solve_nldf)
Note: this function uses optional parameters to define choices in the problem specification and in the details of the algorithm. If you wish to use default
settings for all of the optional parameters, you need only read Sections 1 to 10 of this document. If, however, you wish to reset some or all of the settings please refer to Section 11 for a detailed description of the algorithm and to Section 12 for a detailed description of the specification of the optional parameters.
1
Purpose
e04gnc is a solver from the NAG optimization modelling suite for general nonlinear datafitting problems with constraints. Various loss and regularization functions are supported.
2
Specification
The function may be called by the names: e04gnc or nag_opt_handle_solve_nldf.
3
Description
e04gnc solves a datafitting problem of the form
where

${n}_{\mathrm{var}}$ is the number of decision variables,

${m}_{g}$ is the number of the nonlinear constraints and $g\left(x\right)$, ${l}_{g}$ and ${u}_{g}$ are ${m}_{g}$dimensional vectors,

${m}_{Q}$ is the number of quadratic constraints,

${m}_{B}$ is the number of the linear constraints and $B$ is a ${m}_{B}\times {n}_{\mathrm{var}}$ matrix, ${l}_{B}$ and ${u}_{B}$ are ${m}_{B}$dimensional vectors,
 there are ${n}_{\mathrm{var}}$ box constraints and ${l}_{x}$ and ${u}_{x}$ are ${n}_{\mathrm{var}}$dimensional vectors.
Here,

$x$ is an ${n}_{\mathrm{var}}$dimensional vector representing the model parameters,

$\chi $ is the loss function,

$\psi $ is the regularization function,

$\rho $ is the regularization coefficient,

${n}_{\mathrm{res}}$ is the number of residuals and ${r}_{i}$ is the $i$th residual, which is defined as
where
$\varphi ({t}_{i};x)$ is the predicted value of the
$i$th data point, given
$x$. For the
$i$th data point,
${y}_{i}$ and
${t}_{i}$ are the observed values of the independent and dependant variables respectively.
The available loss and regularization function types are summarized in
Table 1, where
$d$ is the function parameter and
$I\left(L\right)$ denotes an indicator function taking the value
$1$ if the logical expression
$L$ is true and
$0$ otherwise. Loss function and regularization types can be specified by optional parameters
${\mathbf{NLDF\; Loss\; Function\; Type}}$ and
${\mathbf{Reg\; Term\; Type}}$, respectively. For example, set
${\mathbf{NLDF\; Loss\; Function\; Type}}=\mathrm{LINF}$ and
${\mathbf{Reg\; Term\; Type}}=\mathrm{L2}$ to use
${l}_{\infty}$norm loss function with
${l}_{2}$norm (Ridge) regularization. See
Section 11 for more details on the loss functions.
Table 1
Choices for the loss and regularization function types.
Loss function 
$\mathit{\chi}\left({\mathit{r}}_{\mathit{i}}\right)$ 
${\mathbf{NLDF\; Loss\; Function\; Type}}$ 
${l}_{2}$norm 
${r}_{i}^{2}$ 
$\mathrm{L2}$ 
${l}_{1}$norm 
$\left{r}_{i}\right$ 
$\mathrm{L1}$ 
${l}_{\infty}$norm 
$\underset{1\le j\le {n}_{\mathrm{res}}}{\mathrm{max}}}\phantom{\rule{0.25em}{0ex}}\left{r}_{j}\right/{n}_{\mathrm{res}$

$\mathrm{LINF}$ 
Huber (see (7)) 
$\{\begin{array}{ll}0.5*{r}_{i}^{2}& \text{if}\left{r}_{i}\right<d\\ d*(\left{r}_{i}\right0.5*d)& \text{otherwise}\end{array}$

$\mathrm{HUBER}$ 
Cauchy (see (4)) 
$\mathrm{ln}(1+{({r}_{i}/d)}^{2})$

$\mathrm{CAUCHY}$ 
Atan 
$\mathrm{arctan}\left({r}_{i}^{2}\right)$

$\mathrm{ATAN}$ 
SmoothL1 (see (8)) 
$\{\begin{array}{ll}0.5*{r}_{i}^{2}/d& \text{if}\left{r}_{i}\right<d\\ \left{r}_{i}\right0.5*d& \text{otherwise}\end{array}$

$\mathrm{SMOOTHL1}$ 
Quantile (see (9)) 
${r}_{i}*(d{I}_{({r}_{i}<0)})$

$\mathrm{QUANTILE}$ 
Regularization 
$\mathit{\psi}\left({\mathit{x}}_{\mathit{i}}\right)$ 
${\mathbf{Reg\; Term\; Type}}$ 
Lasso (${l}_{1}$norm) 
$\left{x}_{i}\right$ 
$\mathrm{L1}$ 
Ridge (${l}_{2}$norm) 
${x}_{i}^{2}$ 
$\mathrm{L2}$ 
e04gnc serves as a solver for problems stored as a
handle. The
handle points to an internal data structure which defines the problem and serves as a means of communication for functions in the NAG optimization modelling suite.
After the
handle has been initialized (e.g.,
e04rac has been called),
e04rmc can be used to add a model and define its residual sparsity structure.
e04rsc and
e04rtc may be used to set or modify quadratic constraints. Linear constraints
${l}_{B}$,
$B$,
${u}_{B}$ are handled by
e04rjc. Variable box bounds
${l}_{x}$ and
${u}_{x}$ can be specified with
e04rhc, and
e04rkc can set or modify nonlinear constraints. Once the problem is fully described, the
handle may be passed to the solver
e04gnc. When the
handle is no longer needed,
e04rzc should be called to destroy it and deallocate the memory held within. See
Section 4.1 in the
E04 Chapter Introduction for more details about the NAG optimization modelling suite.
Nonlinear Programming (NLP) solvers
e04kfc and
e04stc are used as solver engines by
e04gnc, which defines the selected loss function and regularization, then transforms the problem into standard form that the NLP solvers allow. For best performance, when the objective function
$f\left(x\right)$ is differentiable and without any constraint other than simple bound constraints,
e04kfc is used. For nondifferentiable objective functions or cases where constraints other than simple variable bounds are present,
e04stc is used. See
Section 11 in
e04kfc and
e04stc for more details on algorithmic details.
The algorithm behaviour can be modified by various optional parameters (see
Section 12) which can be set by
e04zmc and
e04zpc anytime between the initialization of the
handle by e.g.,
e04rac and a call to the solver. Once the solver has finished, options may be modified for the next solve. The solver may be called repeatedly with various starting points and/or optional parameters. Option getter
e04znc can be called to retrieve the current value of any option.
4
References
None.
5
Arguments

1:
$\mathbf{handle}$ – void *
Input

On entry: the handle to the problem. It needs to be initialized (e.g., by
e04rac) and to hold a problem formulation compatible with
e04gnc. It
must not be changed between calls to the NAG optimization modelling suite.

2:
$\mathbf{lsqfun}$ – function, supplied by the user
External Function

lsqfun must evaluate the value of the nonlinear residuals,
${r}_{i}\left(x\right)\u2254{y}_{i}\varphi ({t}_{i};x),i=1,\dots ,{n}_{\mathrm{res}}$, at a specified point
$x$.
The specification of
lsqfun is:

1:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

2:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – const double
Input

On entry: $x$, the vector of variable values at which the residuals, ${r}_{i}$, are to be evaluated.

3:
$\mathbf{nres}$ – Integer
Input

On entry: ${n}_{\mathrm{res}}$, the current number of residuals in the model.

4:
$\mathbf{rx}\left[{\mathbf{nres}}\right]$ – double
Output

On exit: the value of the residual vector, $r\left(x\right)$, evaluated at $x$.

5:
$\mathbf{inform}$ – Integer *
Input/Output

On entry: a nonnegative value.
On exit: may be used to indicate that some residuals could not be computed at the requested point. This can be done by setting
inform to a negative value. The solver will attempt a rescue procedure and request an alternative point. If the rescue procedure fails, the solver will exit with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_USER_NAN.

6:
$\mathbf{comm}$ – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to
lsqfun.
 user – double *
 iuser – Integer *
 p – Pointer
The type Pointer will be
void *. Before calling
e04gnc you may allocate memory and initialize these pointers with various quantities for use by
lsqfun when called from
e04gnc (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).

3:
$\mathbf{lsqgrd}$ – function, supplied by the user
External Function

lsqgrd evaluates the residual gradients,
$\nabla {r}_{i}\left(x\right)$, at a specified point
$x$.
The specification of
lsqgrd is:

1:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

2:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – const double
Input

On entry: $x$, the vector of variable values at which the
residual gradients, $\nabla {r}_{i}\left(x\right)$, are to be evaluated.

3:
$\mathbf{nres}$ – Integer
Input

On entry: ${n}_{\mathrm{res}}$, the current number of residuals in the model.

4:
$\mathbf{nnzrd}$ – Integer
Input

On entry: the number of nonzeros in the first derivative matrix.
If
${\mathbf{isparse}}=0$
in the call to
e04rmc (recommended use for
e04gnc) then
${\mathbf{nnzrd}}={\mathbf{nvar}}*{\mathbf{nres}}$.

5:
$\mathbf{rdx}\left[\mathit{dim}\right]$ – double
Input/Output

On entry: the elements should only be assigned and not referenced.
On exit: the vector containing the nonzero residual gradients
evaluated at
$x$,
where
The elements must be stored in the same order as the defined sparsity pattern
provided in the call to
e04rmc.

6:
$\mathbf{inform}$ – Integer *
Input/Output

On entry: a nonnegative value.
On exit: may be used to indicate that the residual gradients could not be computed at the requested point. This can be done by setting
inform to a negative value. The solver will attempt a rescue procedure and request an alternative point. If the rescue procedure fails, the solver will exit with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_USER_NAN.

7:
$\mathbf{comm}$ – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to
lsqgrd.
 user – double *
 iuser – Integer *
 p – Pointer
The type Pointer will be
void *. Before calling
e04gnc you may allocate memory and initialize these pointers with various quantities for use by
lsqgrd when called from
e04gnc (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).

4:
$\mathbf{confun}$ – function, supplied by the user
External Function

confun must calculate the values of the
${m}_{g}$element vector
${g}_{i}\left(x\right)$ of nonlinear constraint functions at a specified value of the
${n}_{\mathrm{var}}$element vector of
$x$ variables. If there are no nonlinear constraints then
confun will never be called by
e04gnc and may be specified as
NULLFN.
The specification of
confun is:

1:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

2:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – const double
Input

On entry: the vector $x$ of variable values at which the constraint functions are to be evaluated.

3:
$\mathbf{ncnln}$ – Integer
Input

On entry:
${m}_{g}$, the number of nonlinear constraints, as specified in an earlier call to
e04rkc.

4:
$\mathbf{gx}\left[\mathit{dim}\right]$ – double
Output

On exit: the ${m}_{g}$ values of the nonlinear constraint functions at $x$.

5:
$\mathbf{inform}$ – Integer *
Input/Output

On entry: a nonnegative value.
On exit: must be set to a value describing the action to be taken by the solver on return from
confun. Specifically, if the value is negative, then the value of
gx will be discarded and the solver will either attempt to find a different trial point or terminate immediately with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_USER_NAN; otherwise, the solver will proceed normally.

6:
$\mathbf{comm}$ – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to
confun.
 user – double *
 iuser – Integer *
 p – Pointer
The type Pointer will be
void *. Before calling
e04gnc you may allocate memory and initialize these pointers with various quantities for use by
confun when called from
e04gnc (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).
Note: confun should not return floatingpoint NaN (Not a Number) or infinity values, since these are not handled by
e04gnc. If your code inadvertently
does return any NaNs or infinities,
e04gnc is likely to produce unexpected results.

5:
$\mathbf{congrd}$ – function, supplied by the user
External Function

congrd must calculate the nonzero values of the sparse Jacobian of the nonlinear constraint functions,
$\frac{\partial {g}_{i}}{\partial x}$, at a specified value of the
${n}_{\mathrm{var}}$element vector of
$x$ variables. If there are no nonlinear constraints,
congrd will never be called by
e04gnc and may be specified as
NULLFN.
The specification of
congrd is:

1:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

2:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – const double
Input

On entry: the vector $x$ of variable values at which the Jacobian of the constraint functions is to be evaluated.

3:
$\mathbf{nnzgd}$ – Integer
Input

On entry: is the number of nonzero elements in the sparse Jacobian of the constraint functions, as was set in a previous call to
e04rkc.

4:
$\mathbf{gdx}\left[\mathit{dim}\right]$ – double
Input/Output

On entry: the elements should only be assigned and not referenced.
On exit: the nonzero values of the Jacobian of the nonlinear constraints, in the order specified by
irowgd and
icolgd in an earlier call to
e04rkc.
${\mathbf{gdx}}\left[\mathit{i}1\right]$ will be the gradient
$\frac{\partial {g}_{j}}{\partial {x}_{k}}$, where
$j={\mathbf{irowgd}}\left[\mathit{i}1\right]$ and
$k={\mathbf{icolgd}}\left[\mathit{i}1\right]$.

5:
$\mathbf{inform}$ – Integer *
Input/Output

On entry: a nonnegative value.
On exit: must be set to a value describing the action to be taken by the solver on return from
congrd. Specifically, if the value is negative the solution of the current problem will terminate immediately with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_USER_NAN; otherwise, computations will continue.

6:
$\mathbf{comm}$ – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to
congrd.
 user – double *
 iuser – Integer *
 p – Pointer
The type Pointer will be
void *. Before calling
e04gnc you may allocate memory and initialize these pointers with various quantities for use by
congrd when called from
e04gnc (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).
Note: congrd should not return floatingpoint NaN (Not a Number) or infinity values, since these are not handled by
e04gnc. If your code inadvertently
does return any NaNs or infinities,
e04gnc is likely to produce unexpected results.

6:
$\mathbf{monit}$ – function, supplied by the user
External Function

monit is provided to enable monitoring of the progress of the optimization and, if necessary, to halt the optimization process.
If no monitoring is required,
monit may be specified as
NULLFN.
monit is called at the end of every
$i$th step where
$i$ is controlled by the optional parameter
${\mathbf{NLDF\; Monitor\; Frequency}}$ (if the value is
$0$,
monit is not called).
The specification of
monit is:

1:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

2:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – const double
Input

On entry: the current best point.

3:
$\mathbf{inform}$ – Integer *
Input/Output

On entry: a nonnegative value.
On exit: may be used to request the solver to stop immediately
by setting
inform to a nonzero value in which case it will terminate
with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_USER_STOP; otherwise, the solver will proceed normally.

4:
$\mathbf{rinfo}\left[100\right]$ – const double
Input

On entry: best objective value computed and various indicators (the values are as described in the main argument
rinfo).

5:
$\mathbf{stats}\left[100\right]$ – const double
Input

On entry: solver statistics at monitoring steps or at the end of the current iteration (the values are as described in the main argument
stats).

6:
$\mathbf{comm}$ – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to
monit.
 user – double *
 iuser – Integer *
 p – Pointer
The type Pointer will be
void *. Before calling
e04gnc you may allocate memory and initialize these pointers with various quantities for use by
monit when called from
e04gnc (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).

7:
$\mathbf{nvar}$ – Integer
Input

On entry: ${n}_{\mathrm{var}}$, the current number of decision variables,
$x$, in the model.

8:
$\mathbf{x}\left[{\mathbf{nvar}}\right]$ – double
Input/Output

On entry: ${x}_{0}$, the initial estimates of the variables, $x$.
On exit: the final values of the variables, $x$.

9:
$\mathbf{nres}$ – Integer
Input

On entry: ${n}_{\mathrm{res}}$, the current number of residuals in the model.

10:
$\mathbf{rx}\left[{\mathbf{nres}}\right]$ – double
Output

On exit: the values of the residuals at the final point given in
x.

11:
$\mathbf{rinfo}\left[100\right]$ – double
Output

On exit: objective value and various indicators at monitoring steps or at the end of the final iteration. The measures are:
$0$ 
Objective function value, $f\left(x\right)$ in (1). 
$1$ 
Loss function value, $\sum _{i=1}^{{n}_{\mathrm{res}}}}\chi \left({r}_{i}\left(x\right)\right)$ in (1).

$2$ 
Regularization term value, $\sum _{i=1}^{{n}_{\mathrm{var}}}}\psi \left({x}_{i}\right)$ in (1).

$3$ 
Solution optimality measure. 
$499$ 
Reserved for future use. 

12:
$\mathbf{stats}\left[100\right]$ – double
Output

On exit: solver statistics at monitoring steps or at the end of the final iteration:
$0$ 
Number of iterations performed. 
$1$ 
Total time in seconds spent in the solver. It includes time spent in usersupplied subroutines. 
$299$ 
Reserved for future use. 

13:
$\mathbf{comm}$ – Nag_Comm *

The NAG communication argument (see
Section 3.1.1 in the Introduction to the NAG Library CL Interface).

14:
$\mathbf{fail}$ – NagError *
Input/Output

The NAG error argument (see
Section 7 in the Introduction to the NAG Library CL Interface).
e04gnc returns with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_NOERROR if the iterates have converged to a point
$x$ that satisfies the convergence criteria described in
Section 7.
6
Error Indicators and Warnings
 NE_ALLOC_FAIL

Dynamic memory allocation failed.
See
Section 3.1.2 in the Introduction to the NAG Library CL Interface for further information.
 NE_BAD_PARAM

On entry, argument $\u27e8\mathit{\text{value}}\u27e9$ had an illegal value.
 NE_FAILED_START

The current starting point is unusable.
Either
inform was set to a negative value within the usersupplied functions
lsqfun,
lsqgrd, or an Infinity or NaN was detected in values returned from them.
 NE_HANDLE

The supplied
handle does not define a valid handle to the data structure for the NAG optimization modelling suite. It has not been properly initialized or it has been corrupted.
 NE_INTERNAL_ERROR

An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact
NAG for assistance.
See
Section 7.5 in the Introduction to the NAG Library CL Interface for further information.
 NE_MAYBE_INFEASIBLE

The solver detected an infeasible problem. This indicates that the problem may be infeasible or at least that the algorithm is stuck at a locally infeasible point. If you believe that the problem is feasible, it might help to start the solver from a different point.
 NE_NO_IMPROVEMENT

The solver was terminated because no further progress could be achieved.
This can indicate that the solver is calculating very small step sizes and is making very little progress. It could also indicate that the problem has been solved to the best numerical accuracy possible given the current scaling.
 NE_NO_LICENCE

Your licence key may have expired or may not have been installed correctly.
See
Section 8 in the Introduction to the NAG Library CL Interface for further information.
 NE_NOT_IMPLEMENTED

e04gnc is not available in this implementation.
 NE_NULL_ARGUMENT

The problem requires the
confun values. Please provide a proper
confun function.
The problem requires the
congrd derivatives. Please provide a proper
congrd function.
 NE_PHASE

The problem is already being solved.
 NE_REF_MATCH

On entry,
${\mathbf{nres}}=\u27e8\mathit{\text{value}}\u27e9$, expected
$\mathrm{value}=\u27e8\mathit{\text{value}}\u27e9$.
Constraint:
nres must match the current number of residuals defined in the
handle.
On entry,
${\mathbf{nvar}}=\u27e8\mathit{\text{value}}\u27e9$, expected
$\mathrm{value}=\u27e8\mathit{\text{value}}\u27e9$.
Constraint:
nvar must match the current number of variables of the model in the
handle.
 NE_SETUP_ERROR

This solver does not support the model defined in the handle.
 NE_SUBPROBLEM

The solver terminated after an error in the step computation. This message is printed if the solver is unable to compute a search direction, despite several attempts to modify the iteration matrix.
The solver terminated after failure during line search. This could happen if the transformed problem is highly degenerate, does not satisfy the constraint qualification, or if the usersupplied code provides incorrect derivative information.
The solver terminated with not enough degrees of freedom. This indicates that the problem, as specified, has too few degrees of freedom. This can happen if there are too many equality constraints, or if there are too many fixed variables.
 NE_TIME_LIMIT

The solver terminated after the maximum time allowed was exhausted.
Maximum number of seconds exceeded. Use optional parameter ${\mathbf{Time\; Limit}}$ to change the limit.
 NE_TOO_MANY_ITER

Maximum number of iterations reached.
 NE_USER_NAN

Invalid number detected in user function.
Either inform was set to a negative value within the usersupplied functions lsqfun, lsqgrd, confun, congrd, or an Infinity or NaN was detected in values returned from them.
 NE_USER_STOP

User requested termination during a monitoring step.
inform was set to a negative value in
monit.
 NW_NOT_CONVERGED

Problem was solved to an acceptable level; full accuracy was not achieved.
This indicates that the algorithm detected a sequence of very small reductions in the objective function value and is unable to reach a point satisfying the requested optimality tolerance. This may happen if the desired tolerances are too small for the current problem, or if the input data is badly scaled.
7
Accuracy
The accuracy of the solution is determined by optional parameters ${\mathbf{NLDF\; Stop\; Tolerance}}$.
In the case where both loss function and regularization are differentiable, and with only simple bound constraints, if
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_NOERROR on exit, the returned point satisfies the firstorder optimality conditions defined by
(6) and
(7) in
e04kfc to the requested accuracy. If the loss or regularization functions are nondifferentiable, or the model defines linear, quadratic or general nonlinear constraint, the model is transformed and solved by
e04stc. In this case, if
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_NOERROR on exit, the returned point satisfies the Karush–Kuhn–Tucker (KKT) condition defined by
(10) in
e04stc to the requested accuracy.
8
Parallelism and Performance
Background information to multithreading can be found in the
Multithreading documentation.
e04gnc is not threaded in any implementation.
9.1
Description of the Printed Output
The solver can print information to give an overview of the problem and the progress of the computation. The output may be sent to two independent file ID which are set by optional parameters ${\mathbf{Print\; File}}$ and ${\mathbf{Monitoring\; File}}$. Optional parameters ${\mathbf{Print\; Level}}$, ${\mathbf{Print\; Options}}$, ${\mathbf{Monitoring\; Level}}$ and ${\mathbf{Print\; Solution}}$ determine the exposed level of detail. This allows, for example, a detailed log file to be generated while the condensed information is displayed on the screen.
By default (${\mathbf{Print\; File}}=6$, ${\mathbf{Print\; Level}}=2$), the following sections are printed to the standard output:
 Header
 Optional parameters list (if ${\mathbf{Print\; Options}}=\mathrm{YES}$)
 Problem statistics
 Iteration log
 Summary
 Solution (if ${\mathbf{Print\; Solution}}=\mathrm{YES}$)
Header
The header is a message indicating the start of the solver. It should look like:

E04GN, Nonlinear DataFitting

Optional parameters list
If
${\mathbf{Print\; Options}}=\mathrm{YES}$, a list of the optional parameters and their values is printed before the problem statistics. The list shows all options of the solver, each displayed on one line. Each line contains the option name, its current value and an indicator for how it was set. The options unchanged from their defaults are noted by ‘d’ and the ones you have set are noted by ‘U’. Note that the output format is compatible with the file format expected by
e04zpc. The output looks similar to:
Begin of Options
Nldf Loss Function Type = Smoothl1 * U
Nldf Huber Function Width = 1.00000E+00 * d
Nldf Cauchy Function Sharpness= 1.00000E+00 * d
Nldf Smoothl1 Function Width = 1.00000E+00 * d
Nldf Quantile Parameter = 5.00000E01 * d
Nldf Iteration Limit = 10000000 * d
Nldf Stop Tolerance = 1.00000E06 * d
Nldf Monitor Frequency = 0 * d
End of Options
Problem statistics
If
${\mathbf{Print\; Level}}\ge 2$, statistics on the problem are printed, for example:
Problem Statistics
No of variables 6
linear 2
nonlinear 4
free (unconstrained) 4
bounded 2
No of lin. constraints 1
nonzeroes 2
No of quad.constraints 0
No of nln. constraints 4
nonzeroes 3
Loss function SmoothL1
No of residuals 24
Regularization L1 Norm
Iteration log
If
${\mathbf{Print\; Level}}\ge 2$, the solver will print a summary line for each step. If no regularization function is specified, the output shows the iteration number, the current loss function value and the optimality measure. The output looks as follows:

it lossfun  optim

1 1.05863E+02 3.25E+02
2 3.51671E+01 9.03E+01
3 2.79091E+01 4.95E+01
4 2.63484E+01 1.43E+01
5 2.30795E+01 2.59E+01
If you specify a regularization type via
${\mathbf{Reg\; Term\; Type}}$, two more columns will be printed out showing the regularization function value and the objective function value
$f\left(x\right)$ defined in
(1). It might look as follows:

it objfun  lossfun  reg  optim

1 2.11489E+01 2.02833E+01 8.65649E01 1.27E+02
2 2.28405E+01 2.21698E+01 6.70710E01 9.47E+01
3 2.30351E+01 2.23599E+01 6.75246E01 1.62E+01
4 2.22836E+01 2.15675E+01 7.16133E01 1.32E+01
5 2.22676E+01 2.15522E+01 7.15411E01 3.80E+00
6 2.21776E+01 2.14746E+01 7.02983E01 3.92E01
7 2.22075E+01 2.15060E+01 7.01510E01 6.08E01
Summary
Once the solver finishes, a summary is produced:

Status: converged, an optimal solution found

Final objective value 2.216883E+01
Final loss function value 2.146563E+01
Final regularization value 7.031993E01
Solver stopping precision 1.068791E08
Iterations 16
Optionally, if
${\mathbf{Stats\; Time}}=\mathrm{YES}$, the timings are printed:
Total time 0.19 sec
Solution
If
${\mathbf{Print\; Solution}}=\mathrm{YES}$, the values of the primal variables are printed. It might look as follows:
Primal variables:
idx Lower bound Value Upper bound
1 1.00000E+00 9.69201E02 inf
2 0.00000E+00 7.95110E01 1.00000E+00
10
Example
This example demonstrates how to define and solve a nonlinear regression problem using both least squares and robust regression. The regression problem consists of
${\mathbf{nres}}=24$ observations,
$(t,y)$, to be fitted over the
${\mathbf{nvar}}=2$ parameter model
and the residuals for the regression are
where
The data above is generated by setting ${x}_{1}=0.1$ and ${x}_{2}=0.8$ with added random noise. Data point $({t}_{i},{y}_{i})$ for $i=4,12,16,20$ are set to be outliers.
The following bounds and general linear constraints are defined on the variables
Setting the initial guess to be ${x}_{0}=(0.3,0.7)$, the problem is first solved using least squares loss function and the expected solution is ${x}^{*}=(0.0944,0.7740)$.
Then the loss function type is switched to SmoothL1 and the problem is solved again. The expected solution is ${x}^{*}=(0.0969,0.7951)$, which is an improvement over least squares fit for outlier handling.
10.1
Program Text
10.2
Program Data
None.
10.3
Program Results
11
Algorithmic Details
e04gnc will use the provided model, model derivative, and data to internally construct an optimization problem, and pass it off to the appropriate nonlinear programming solver. Depending on the choice of loss and regularization functions, as well as the types of constraints present, either a firstorder active set method solver (e.g.,
e04kfc) or a general nonlinear programming solver (e.g.,
e04stc) will be called to maximize performance.
11.1
Loss Function and Regularization Types
Loss function and regularization types can be specified by optional parameters
${\mathbf{NLDF\; Loss\; Function\; Type}}$ and
${\mathbf{Reg\; Term\; Type}}$, respectively. The definitions of the available loss and regularization functions are:

•L$2$ loss function:
L$2$ loss involves ${l}_{2}$norm which defines the least square errors. This is the default loss function used by e04gnc.

•L$1$ loss function:
L$1$ loss involves ${l}_{1}$norm which defines the least absolute deviation. It is robust against outliers, while still allowing them to impact the search direction to some degree. Outliers are not handled by L$1$ loss as robustly as e.g., the Atan loss function.

•Cauchy loss function:
where ${d}_{\mathrm{cau}}$ represents the optional parameter ${\mathbf{NLDF\; Cauchy\; Function\; Sharpness}}$.
Cauchy loss is robust against outliers, more so than L$1$ loss but less than Atan loss. An advantage of using this function is that the residual derivatives (Jacobian) approach $0$ for large input, but slower than Atan loss by a factor proportional to ${r}_{i}{\left(x\right)}^{2}$. The parameter ${d}_{\mathrm{cau}}$ controls how sharp the Cauchy function is, with large ${d}_{\mathrm{cau}}$ resulting in a more flattened curve.

•Atan loss function:
Atan tends to
$\pi /2$ as input tends to infinity, and its derivatives tend to
$0$. This means extreme outliers will have a negligible effect on the search direction compared to nonoutliers. It also means it is important to ensure the data is well scaled and that the starting point is a reasonable guess at the true solution, if possible.

•Linf loss function:
Linf loss involves
${l}_{\infty}$norm which leads to the term
$\sum _{i=1}^{{n}_{\mathrm{res}}}}\chi \left({r}_{i}\left(x\right)\right)$ in the objective function evaluating to
${\mathrm{max}}_{j}\left{r}_{j}\left(x\right)\right$. This will typically be a much smaller number than for other loss functions, since only one residual is contributing. This also means that the regularization function (if present) will have a relatively larger effect; using a lower regularization coefficient can balance this out.

•Huber loss function:
where
${d}_{\mathrm{hub}}$ represents the optional parameter
${\mathbf{NLDF\; Huber\; Function\; Width}}$.
Huber loss is robust against outliers, since it increases linearly with $\left{r}_{i}\left(x\right)\right\ge {d}_{\mathrm{hub}}$. It is also differentiable at $0$. The loss function behaviour depends on the chosen value of ${d}_{\mathrm{hub}}$; the value of ${d}_{\mathrm{hub}}$ must be smaller than the difference between the outliers and the model for them to fall within the linear part of the Huber function. Increasing the parameter also increases the height and steepness of the linear part; this should be considered when comparing the final objective function value for different values of ${d}_{\mathrm{hub}}$, as well as the relative effect of regularization.

•SmoothL$1$ loss function:
where
${d}_{\mathrm{sl}1}$ represents the optional parameter
${\mathbf{NLDF\; SmoothL1\; Function\; Width}}$.
This loss function is equal to Huber loss weighted by a factor of $1/{d}_{\mathrm{sl}1}$. The linear part's gradient is unaffected by ${d}_{\mathrm{sl}1}$, so the relative effect of regularization will be similar over different values of ${d}_{\mathrm{sl}1}$. Functionally it is similar to L$1$ loss if ${d}_{\mathrm{sl}1}$ is small, and L$2$ loss if ${d}_{\mathrm{sl}1}$ is very large.

•Quantile loss function:
where
and
${d}_{\mathrm{qnt}}$ represents the optional parameter
${\mathbf{NLDF\; Quantile\; Parameter}}$. Quantile loss function is similar to L
$1$ loss, but adjustment of the parameter
${d}_{\mathrm{qnt}}$ can ‘tilt’ it. For example,
${d}_{\mathrm{qnt}}=0.95$ will lead to the model being fitted above more with respect to L
$1$ loss of the data, since negative residuals contribute lower loss than positive ones.
${d}_{\mathrm{qnt}}=0.05$ will have the opposite effect; the model will be fitted below more of the data, since positive residuals give lower loss than negative ones.

•L$1$ regularization:

•L$2$ regularization
11.2
Optimization Problem Transformations
e04gnc uses auxiliary variables and constraints to implement nondifferentiable loss functions (L$1$, Quantile, Linf) and regularization functions (L$1$). For any method that uses loss function derivatives to compute a search direction, a loss function which is not differentiable will cause problems, especially if it is nondifferentiable at its minima. To solve this, we transform a model with nondifferentiable loss or regularization functions into a new model with differentiable constraints and a differentiable objective function.
Consider the case where we select L
$1$ loss and turn off regularization:
This is equivalent to the problem:
where the
${z}_{i}>0$ are
${n}_{\mathrm{res}}$ new auxiliary variables.
Each ${z}_{i}$ serves as an upper bound to $\left{r}_{i}\left(x\right)\right$, so that a minimum of $\sum _{i=1}^{{n}_{\mathrm{res}}}}{z}_{i$ in the search space comprising $x$ and ${z}_{i}$, is also a minimum of $\sum _{i=1}^{{n}_{\mathrm{res}}}}\left{r}_{i}\left(x\right)\right$.
Similarly for Quantile loss with no regularization, we transform the problem:
into the equivalent model:
For Linf loss with no regularization, we transform the model:
into the model:
If regularization is present, we transform it separately from the loss function if the regularization is nondifferentiable. For example, the problem
is transformed to:
where the
${v}_{i}$ are the auxiliary variables for the regularization.
12
Optional Parameters
Several optional parameters in e04gnc define choices in the problem specification or the algorithm logic. In order to reduce the number of formal arguments of e04gnc, these optional parameters have associated default values that are appropriate for most problems. Therefore, you need only specify those optional parameters whose values are to be different from their default values.
The remainder of this section can be skipped if you wish to use the default values for all optional parameters.
The optional parameters can be changed by calling
e04zmc anytime between the initialization of the handle and the call to the solver. Modification of the optional parameters during intermediate monitoring stops is not allowed. Once the solver finishes, the optional parameters can be altered again for the next solve.
The option values may be retrieved by
e04znc.
The following is a list of the optional parameters available. A full description of each optional parameter is provided in
Section 12.1.
12.1
Description of the Optional Parameters
For each option, we give a summary line, a description of the optional parameter and details of constraints.
The summary line contains:
 the keywords, where the minimum abbreviation of each keyword is underlined;
 a parameter value,
where the letters $a$, $i$ and $r$ denote options that take character, integer and real values respectively;
 the default value, where the symbol $\epsilon $ is a generic notation for machine precision (see X02AJC).
All options accept the value $\mathrm{DEFAULT}$ to return single options to their default states.
Keywords and character values are case and white space insensitive.
This special keyword may be used to reset all optional parameters to their default values. Any value given with this keyword will be ignored.
NLDF Iteration Limit  $i$  Default $=10000000$ 
The maximum number of iterations to be performed by
e04gnc. If this limit is reached, then the solver will terminate with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_TOO_MANY_ITER.
Constraint: ${\mathbf{NLDF\; Iteration\; Limit}}\ge 1$.
NLDF Monitor Frequency  $i$  Default $=0$ 
If
${\mathbf{NLDF\; Monitor\; Frequency}}>0$, the usersupplied function
monit will be called at the end of every
$i$th step for monitoring purposes.
Constraint: ${\mathbf{NLDF\; Monitor\; Frequency}}\ge 0$.
NLDF Stop Tolerance  $r$  Default $=\mathrm{max}\phantom{\rule{0.125em}{0ex}}({10}^{\mathrm{6}},\sqrt{\epsilon})$ 
This parameter sets the value of ${\epsilon}_{\mathrm{tol}}$ which specifies the tolerance for the optimality measure.
When both loss function and regularization are differentiable, and with only simple bound constraints, the optimality measures are defined by
(6) and
(7) in
e04kfc and
${\mathbf{NLDF\; Stop\; Tolerance}}$ is passed to the solver
e04kfc as
${\mathbf{FOAS\; Stop\; Tolerance}}$.
When any of the loss function or regularization is nondifferentiable, or there presents linear, quadratic or general nonlinear constraint, the optimality measure is defined by
(10) in
e04stc and
${\mathbf{NLDF\; Stop\; Tolerance}}$ is passed to the solver
e04stc as
${\mathbf{Stop\; Tolerance\; 1}}$.
Constraint: $0\le {\epsilon}_{\mathrm{tol}}<1$.
NLDF Loss Function Type  $a$  Default $=$$\mathrm{L2}$ 
This parameter sets the loss function type used in the objective function.
Constraint: ${\mathbf{NLDF\; Loss\; Function\; Type}}=\mathrm{HUBER}$, $\mathrm{L2}$, $\mathrm{CAUCHY}$, $\mathrm{ATAN}$, $\mathrm{SMOOTHL1}$, $\mathrm{LINF}$, $\mathrm{L1}$ or $\mathrm{QUANTILE}$.
Reg Term Type  $a$  Default $=$$\mathrm{OFF}$ 
This parameter sets the regularization function type used in the objective function.
Note: if there is no residual in the model, regularization will be turned off.
Constraint: ${\mathbf{Reg\; Term\; Type}}=\mathrm{OFF}$, $\mathrm{L2}$ or $\mathrm{L1}$.
NLDF Huber Function Width  $r$  Default $\text{}=1.0$ 
Sets the parameter
${d}_{\mathrm{hub}}$ defined in the Huber loss function
(7).
Constraint: ${\mathbf{NLDF\; Huber\; Function\; Width}}>0$.
NLDF Cauchy Function Sharpness  $r$  Default $\text{}=1.0$ 
Sets the parameter
${d}_{\mathrm{cau}}$ defined in the Cauchy loss function
(4).
Constraint: ${\mathbf{NLDF\; Cauchy\; Function\; Sharpness}}>0$.
NLDF SmoothL1 Function Width  $r$  Default $\text{}=1.0$ 
Sets the parameter
${d}_{\mathrm{sl}1}$ defined in the SmoothL1 loss function
(8).
Constraint: ${\mathbf{NLDF\; SmoothL1\; Function\; Width}}>0$.
NLDF Quantile Parameter  $r$  Default $\text{}=0.5$ 
Sets the parameter
${d}_{\mathrm{qnt}}$ defined in the Quantile loss function
(9).
Constraint: $0<{\mathbf{NLDF\; Quantile\; Parameter}}<1$.
Reg Coefficient  $r$  Default $\text{}=1.0$ 
Sets the regularization coefficient
$\rho $ in the definition of the objective function
(1).
Note: if set to
$0$, regularization will be turned off.
Constraint: ${\mathbf{Reg\; Coefficient}}\ge 0$.
Infinite Bound Size  $r$  Default $\text{}={10}^{20}$ 
This defines the ‘infinite’ bound $\mathit{bigbnd}$ in the definition of the problem constraints. Any upper bound greater than or equal to $\mathit{bigbnd}$ will be regarded as $+\infty $ (and similarly any lower bound less than or equal to $\mathit{bigbnd}$ will be regarded as $\infty $). Note that a modification of this optional parameter does not influence constraints which have already been defined; only the constraints formulated after the change will be affected.
Constraint: ${\mathbf{Infinite\; Bound\; Size}}\ge 1000$.
Monitoring File  $i$  Default $\text{}=\mathrm{1}$ 
(See
Section 3.1.1 in the Introduction to the NAG Library CL Interface for further information on NAG data types.)
If
$i\ge 0$, the
Nag_FileID number (as returned from
x04acc) for the secondary (monitoring) output. If
${\mathbf{Monitoring\; File}}=\mathrm{1}$, no secondary output is provided. The information output to this file ID is controlled by
${\mathbf{Monitoring\; Level}}$.
Constraint: ${\mathbf{Monitoring\; File}}\ge \mathrm{1}$.
Monitoring Level  $i$  Default $=4$ 
This parameter sets the amount of information detail that will be printed by the solver to the secondary output. The meaning of the levels is the same as for ${\mathbf{Print\; Level}}$.
Constraint: $0\le {\mathbf{Monitoring\; Level}}\le 5$.
Print File  $i$  Default
$=\mathrm{Nag\_FileID\; number\; associated\; with\; stdout}$

(See
Section 3.1.1 in the Introduction to the NAG Library CL Interface for further information on NAG data types.)
If
$i\ge 0$, the
Nag_FileID number (as returned from
x04acc,
stdout as the default) for the primary output of the solver. If
${\mathbf{Print\; File}}=\mathrm{1}$, the primary output is completely turned off independently of other settings. The information output to this unit is controlled by
${\mathbf{Print\; Level}}$.
Constraint: ${\mathbf{Print\; File}}\ge \mathrm{1}$.
Print Level  $i$  Default $=2$ 
This parameter defines how detailed information should be printed by the solver to the primary and secondary output.
$\mathit{i}$ 
Output 
$0$ 
No output from the solver. 
$1$ 
The Header and Summary. 
$2$, $3$, $4$, $5$ 
Additionally, the Iteration log. 
Constraint: $0\le {\mathbf{Print\; Level}}\le 5$.
Print Options  $a$  Default $=\mathrm{YES}$ 
If ${\mathbf{Print\; Options}}=\mathrm{YES}$, a listing of optional parameters will be printed to the primary output and is always printed to the secondary output.
Constraint: ${\mathbf{Print\; Options}}=\mathrm{YES}$ or $\mathrm{NO}$.
Print Solution  $a$  Default $=\mathrm{NO}$ 
If ${\mathbf{Print\; Solution}}=\mathrm{YES}$ or $\mathrm{X}$, the final values of the primal variables are printed on the primary and secondary outputs.
Constraint: ${\mathbf{Print\; Solution}}=\mathrm{YES}$, $\mathrm{NO}$ or $\mathrm{X}$.
Stats Time  $a$  Default $=\mathrm{NO}$ 
This parameter turns on timing. This might be helpful for a choice of different solving approaches. It is possible to choose between CPU and wall clock time. Choice $\mathrm{YES}$ is equivalent to $\mathrm{WALL\; CLOCK}$.
Constraint: ${\mathbf{Stats\; Time}}=\mathrm{YES}$, $\mathrm{NO}$, $\mathrm{CPU}$ or $\mathrm{WALL\; CLOCK}$.
Time Limit  $r$  Default $\text{}={10}^{6}$ 
A limit to the number of seconds that the solver can use to solve one problem. If at the end of an iteration this limit is exceeded, the solver will terminate with
${\mathbf{fail}}\mathbf{.}\mathbf{code}=$ NE_TIME_LIMIT.
Constraint: ${\mathbf{Time\; Limit}}>0$.