This chapter is concerned with utility operations used in conjunction with the NAG AD Library. Details specific to the NAG AD Library can be found in the Introduction to the NAG AD Library.
All routines in the NAG AD Library have, as first argument in their interfaces, a handle to a configuration data object which is consistently named throughout the NAG AD Library as ad_handle. This argument stores configuration details used in the calculation of algorithmic derivatives. Further details and examples on the use of ad_handle can be found in the Introduction to the NAG AD Library.
Chapter X10 provides facilities for:
(i)Creating or removing a configuration data object
(ii)Setting or getting particular configuration details
(iii)Creating a callback object
(iv)Setting or getting callback object configuration details
(v)Inserting a callback location into a callback object
(vi)Reading from and writing to a callback object
2Background to the Problems
2.1Configuration Data Objects
There are times when information that affects the form of computation performed by NAG AD Library routines or supplied procedure arguments needs to be communicated through the routine and supplied procedure interfaces.
The configuration data object, ad_handle is used, in particular, to pass the AD computational mode to be used (see Section 2.2). This argument always appears as the first argument to NAG AD Library computational routines.
When a routine contains a procedure argument that has active type arguments in its interface, ad_handle will also be the first argument of that interface. In addition to computational mode, ad_handle may also contain the form of operation to be performed by a procedure argument.
There is one other data object used by the NAG AD Library. This is the callback data object, which is specific to the calculation of adjoints and is used to: pass data from a procedure argument to a fixed interface companion callback; insert the location of the companion callback; and force the internal execution of the companion callback at the appropriate point in the evaluation of adjoints. This object is consistently named throughout the NAG AD Library as cb_handle.
2.2AD Computational Modes
There are two basic types of AD computational mode.
(i)symbolic mode (nagad_symbolic)
This refers to the case where the differential of the primal calculation has been specifically coded, and that code is used to calculate, for example, the adjoints or tangents of active variables.
(ii)algorithmic mode (nagad_algorithmic)
This refers to the case where dco (see the Introduction to the NAG AD Library) is used internally through instrumented code to facilitate the calculation of, for example, the adjoints of active variables.
Other modes are possible, for example the following two cases.
(i)hybrid mode
This refers to some combination of algorithmic and symbolic modes, in which some part (though not all) of the differential calculation is performed using symbolic code.
(ii)robust mode
Here the intrumented algorithmic code has been refined by hand to improve various aspects of the computation. This can include: performance improvements through knowledge of the algorithm and through profiling; and reducing the memory footprint through knowledge of the algorithm and checkpointing. Some symbolic calculations may replace some algorithmic steps.
2.3Algorithmic Types
There are three aspects to the algorithmic type:
(i)Direction
This basically means whether the tangent linear derivatives or adjoints are being calculated for active variables.
(ii)Order
The order of differentiation to be performed, i.e., first, second or higher order.
(iii)Precision
Differentiation can be performed in working precision, reduced precision or higher precision. These terms are relative to the precision used in the primal calculation.
There is a separate type for each combination of these three aspects. For example, the algorithmic type codified as a1w refers to the type that calculates adjoints (a) to first order (1) in working precision (w).
2.4Supplied Procedure Arguments (Callbacks)
2.4.1Callbacks in algorithmic mode
Usually this is a very straightforward change of type in the translation from callback in the corresponding primal routine of the NAG Library to the NAG AD Library callback. Overloading / extension of basic arithmetic operations and intrinsic functions means that algorithmic differentiation is automatically performed on simple code containing only arithmetic operations, intrinsic evaluations, or calls to other routines in the NAG AD Library.
Where the operations performed within a supplied procedure argument (callback) are more complex than those described above, the differentiation will have to be performed in a staged manner: primal calculation, differentiation of active output variables with respect to active input (state) variables, and differentiation with respect to user supplied variables (e.g., problem parameters stored in ruser array). The stage to be performed by the supplied procedure is supplied through ad_handle. If computing adjoints and some or all of the operations of a stage need to avoid automatic algorithmic differentation, such operations are performed within a companion callback, which takes the single argument cb_handle. Data required for such operations should be passed, from the procedure argument, through cb_handle, to the companion callback; results of the operations need to be inserted into the global data object for evaluating adjoints via primitive function calls to set values and increment derivatives.
2.4.2Callbacks in symbolic mode
In symbolic mode, the operations performed by a procedure argument are always staged: primal computation, differentiation with respect to active input arguments, differentiation with respect to supplied active variables (e.g., in the array ruser) and, in combining the previous two, differentiation with respect to both state and supplied active variables. When computing the adjoints the operations performed by a stage must be performed through companion callbacks (see Section 2.4.1) to avoid automatic algorithmic differentiation.
3Recommendations on Choice and Use of Available Routines
3.1Creating and Removing Configuration Data Objects
An ad_handle, which is a pointer to an internal data structure, is passed among the routines in the NAG AD Library. It holds configuration details about the AD approach to be used. Routines for creating and removing this ad_handle are as follows.
The computational modes available are routine dependent. All have the default mode of nagad_algorithmic, some have the additional mode of nagad_symbolic, and other modes may become available in the future. Routines for setting and getting the computational modes are as follows.
This is the default mode that will be set when the configuration data object is initialized. In this mode the differentials (e.g., first order adjoints) are calculated by using the following: code created by converting and instrumenting the primal algorithmic code; intrinsic and operator extensions to work with special data types; and dco to perform the differentiation (see Section 2 in the Introduction to the NAG AD Library).
3.2.2The symbolic computational mode
For computational routines in the NAG AD Library where the symbolic computational mode is available, this mode may be selected by calling x10ad_AD_f with mode set to nagad_symbolic prior to calling the computational routine.
The symbolic mode of computation is where code has been written to perform the particular form of differentiation (e.g., first order adjoints) of the primal algorithm.
The list of computational routines at Mark $28.3$ for which the symbolic mode is available is:
Where such a routine contains a procedure argument (callback), then the method of evaluating and providing primal values and their differentials is complicated and involves writing special fixed interface companion callbacks whose location is inserted into a callback data object. The description of this is best described by example (see Section 10 in c05ay_a1w_f and Section 10 in e04gb_a1w_f).
3.3Algorithmic Type
There are currently two algorithmic types, first order adjoints in working precision (a1w) and first order tangents in working precision (t1w). The type coded ‘a1w’ will appear in the short and long names of the routines in this chapter and in the remainder of the NAG AD Library Manual. Four routines in this chapter have corresponding t1w versions: x10aa_t1w_f, x10ab_t1w_f, x10ac_t1w_f, and x10ad_t1w_f. In the remainder of the NAG AD Library Manual, all ‘a1w’ routines have corresponding t1w versions. Interfaces of t1w routines are implied from the documentation of the corresponding ‘a1w’ routine. Where t1w example programs exist, these will be listed in the table of example programs in the ‘a1w’ routine document.
For convenience the algorithmic type p0w refers to zeroth order passive calculations using passive data types. There are no p0w routines in this chapter; however, all routines in the remainder of the NAG AD Library Manual have corresponding p0w versions which differ only in the use of passive types corresponding to the active types of the a1w routines (as listed in Section 2 in the Introduction to the NAG AD Library).
3.4Callback Object Utilities for Adjoints
For routines with one or more supplied routine arguments, the data passed to and from the callback is managed via a callback data object. For each callback supplied, a companion callback is also required to perform the algorithmic differentiation via the callback data object. This companion callback is always a subroutine and always has just one argument: a pointer to the object. In Fortran the interface looks like:
Subroutine adjoint_callback(mode, cb_handle)
Integer (Kind=c_int) :: mode
Type (c_ptr), Value :: cb_handle
End Subroutine
Note that the mode argument is not the same as the callback mode held in the computational data object. It is not currently used, but will become important when further computational modes and further facilities are introduced. When the callback data object is created, it is used to communicate data between the primal callback and the differential callback. Utility routines are provided to read data from and write data to the callback data object as required. The full set of utility routines for managing callbacks are as follows.
Dunford N, Schwartz J T, Bade W G and Bartle R G (1971) Linear Operators Wiley Interscience, New York
Giles M (2008) Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation Springer
Griewank A and Walther A (2008) Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (2nd Edition) SIAM
Hascoët L, Naumann U and Pascual V (2005) ‘To be Recorded’ Analysis in Reverse-Mode Automatic Differentiation Future Generation Computer Systems21(8) 299–304 Elsevier
Naumann U (2012) The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation SIAM
Naumann U, Lotz J, Leppkes K, Towara M (2015) Algorithmic Differentiation of Numerical Methods: Tangent and Adjoint Solvers for Parameterized Systems of Nonlinear Equations ACM Trans. Math. Softw.