NAG Library Chapter Introduction
Automatic Differentiation Utilities
Scope of the Chapter
This chapter is concerned with utility operations used in conjunction with the NAG AD Library. Details specific to the NAG AD Library can be found in the NAG AD Library Introduction
All routines in the NAG AD Library have, as first argument in their interfaces, a handle to a configuration data object
which is consistently named throughout the NAG AD Library as ad_handle
. This argument stores configuration details used in the calculation of algorithmic derivatives. Further details and examples on the use of ad_handle can be found in the NAG AD Library Introduction
provides facilities for:
||Creating or removing a configuration data object
||Setting or getting particular configuration details
||Creating a callback object
||Setting or getting callback object configuration details
||Inserting a callback location into a callback object
||Reading from and writing to a callback object
Background to the Problems
Configuration Data Objects
There are times when information that affects the form of computation performed by NAG AD Library routines or supplied procedure arguments needs to be communicated through the routine and supplied procedure interfaces.
The configuration data object, ad_handle
is used, in particular, to pass the AD computational mode to be used (see Section 2.2
). This argument always appears as the first argument to NAG AD Library computational routines.
When a routine contains a procedure argument that has active type arguments in its interface, ad_handle will also be the first argument of that interface. In addition to computational mode, ad_handle may also contain the form of operation to be performed by a procedure argument.
There is one other data object used by the NAG AD Library. This is the callback data object, which is used to: pass data from a procedure argument to a fixed interface companion callback; insert the location of the companion callback; and force the internal execution of the companion callback at the appropriate point in the evaluation of adjoints. This object is consistently named throughout the NAG AD Library as cb_handle.
AD Computational Modes
There are two basic types of AD computational mode.
||symbolic mode (nagad_symbolic)
This refers to the case where the differential of the primal calculation has been specifically coded, and that code is used to calculate, for example, the adjoints of active variables.
||algorithmic mode (nagad_algorithmic)
This refers to the case where dco
(see the NAG AD Library Introduction
) is used internally through instrumented code to facilitate the calculation of, for example, the adjoints of active variables.
Other modes are possible, for example the following two cases.
This refers to some combination of algorithmic and symbolic modes, in which some part (though not all) of the differential calculation is performed using symbolic code.
Here the intrumented algorithmic code has been refined by hand to improve various aspects of the computation. This can include: performance improvements through knowledge of the algorithm and through profiling; and reducing the memory footprint through knowledge of the algorithm and checkpointing. Some symbolic calculations may replace some algorithmic steps.
There are three aspects to the algorithmic type:
This basically means whether the tangent linear derivatives or adjoints are being calculated for active variables.
The order of differentiation to be performed, i.e., first, second or higher order.
Differentiation can be performed in working precision, reduced precision or higher precision. These terms are relative to the precision used in the primal calculation.
There is a separate type for each combination of these three aspects. For example, the algorithmic type codified as a1w refers to the type that calculates adjoints (a) to first order (1) in working precision (w).
Supplied Procedure Arguments (Callbacks)
Callbacks in algorithmic mode
Usually this is a very straightforward change of type in the translation from callback in the corresponding primal routine of the NAG Library to the NAG AD Library callback. Overloading / extension of basic arithmetic operations and intrinsic functions means that algorithmic differentiation is automatically performed on simple code containing only arithmetic operations, intrinsic evaluations, or calls to other routines in the NAG AD Library.
Where the operations performed within a supplied procedure argument (callback) are more complex than those described above, the differentiation will have to be performed in a staged manner: primal calculation, differentiation of active output variables with respect to active input (state) variables, and differentiation with respect to user supplied variables (e.g., problem parameters stored in ruser array). The stage to be performed by the supplied procedure is supplied through ad_handle. If some or all of the operations of a stage need to avoid automatic algorithmic differentation, such operations are performed within a companion callback, which takes the single argument cb_handle. Data required for such operations should be passed, from the procedure argument, through cb_handle, to the companion callback; results of the operations need to be inserted into the global data object for evaluating adjoints via primitive function calls to set values and increment derivatives.
Callbacks in symbolic mode
In symbolic mode, the operations performed by a procedure argument are always staged: primal computation, differentiation with respect to active input arguments, differentiation with respect to supplied active variables (e.g., in the array ruser) and, in combining the previous two, differentiation with respect to both state and supplied active variables. The operations performed by a stage must be performed through companion callbacks (see Section 2.4.1
) to avoid automatic algorithmic differentiation.
Recommendations on Choice and Use of Available Routines
Creating and Removing Configuration Data Objects
, which is a pointer to an internal data structure, is passed among the routines in the NAG AD Library. It holds configuration details about the AD approach to be used. Routines for creating and removing this ad_handle
are as follows.
- Create a configuration data object.
- Remove a conguration data object.
Setting and Getting the Computational Mode
The computational modes available are routine dependent. All have the default mode of nagad_algorithmic
, some have the additional mode of nagad_symbolic
, and other modes may become available in the future. Routines for setting and getting the computational modes are as follows.
- Set the computational mode.
- Get the computational mode.
The algorithmic computational mode
This is the default mode that will be set when the configuration data object is initialized. In this mode the differentials (e.g., first order adjoints) are calculated by using the following: code created by converting and instrumenting the primal algorithmic code; intrinsic and operator extensions to work with special data types; and dco
to perform the differentiation (see Section 5
in NAG AD Library Introduction).
The symbolic computational mode
For computational routines in the NAG AD Library where the symbolic computational mode is available, this mode may be selected by calling x10ad_a1w_f
set to nagad_symbolic
prior to calling the computational routine.
The symbolic mode of computation is where code has been written to perform the particular form of differentiation (e.g., first order adjoints) of the primal algorithm.
The list of computational routines at Mark 26.2 for which the symbolic mode is available is:
Where such a routine contains a procedure argument (callback), then the method of evaluating and providing primal values and their differentials is complicated and involves writing special fixed interface companion callbacks whose location is inserted into a callback data object
. The description of this is best described by example (see Section 10
and Section 10
There is currently only one algorithmic type, first order adjoints in working precision (a1w). This type is coded as ‘a1w’ and this code will appear in the short and long names of the routines in this chapter and in the remainder of the NAG AD Library Manual.
Callback Object Utilities
For routines with one or more supplied routine arguments, the data passed to and from the callback is managed via a callback data object
. For each callback supplied, a companion callback is also required to perform the algorithmic differentiation via the callback data object
. This companion callback is always a subroutine and always has just one argument: a pointer to the object. In Fortran the interface looks like:
Subroutine adjoint_callback(mode, cb_handle)
Integer (Kind=c_int) :: mode
Type (c_ptr), Value :: cb_handle
Note that the mode
argument is not the same as the callback mode held in the computational data object. It is not currently used, but will become important when further computational modes and further facilities are introduced. When the callback data object is created, it is used to communicate data between the primal callback and the differential callback. Utility routines are provided to read data from and write data to the callback data object
as required. The full set of utility routines for managing callbacks are as follows.
- Create a callback data object
- Insert the location of the AD callback into the callback data object
- Set the callback algorithmic mode.
- Get the callback algorithmic mode.
- Write an integer scalar to the callback data object.
- Write a real (working precision) scalar to the callback data object.
- Write a scalar of type nagad_a1w_w_rtype to the callback data object.
- Read an integer scalar from the callback data object.
- Read a real (working precision) scalar from the callback data object.
- Read a scalar of type nagad_a1w_w_rtype from the callback data object.
Routines Withdrawn or Scheduled for Withdrawal
Dunford N, Schwartz J T, Bade W G and Bartle R G (1971) Linear Operators Wiley Interscience, New York
Giles M (2008) Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation Springer
Griewank A and Walther A (2008) Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiations (2nd Edition) SIAM
Hascoët L, Naumann U and Pascual V (2005) ‘To be Recorded’ Analysis in Reverse-Mode Automatic Differentiation Future Generation Computer Systems 21(8) 299–304 Elsevier
Naumann U (2012) The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation SIAM
Naumann U, Lotz J, Leppkes K, Towara M (2015) Algorithmic Differentiation of Numerical Methods: Tangent and Adjoint Solvers for Parameterized Systems of Nonlinear
Equations ACM Trans. Math. Softw.