NAG Fortran Compiler, Release 7.1: SPMD programming with coarrays [6.2, 7.0]

10.2 SPMD programming with coarrays [6.2, 7.0]

10.2.1 Overview

Fortran 2008 contains an SPMD (Single Program Multiple Data) programming model, where multiple copies of a program, called “images”, are executed in parallel. Special variables called “coarrays” facilitate communication between images.

Release 6.2 of the NAG Fortran Compiler limited execution to a single image, with no parallel execution. Release 7.0 of the NAG Fortran Compiler can execute multiple images in parallel on SMP machines, using Co-SMP technology.

10.2.2 Images

Each image contains its own variables and input/output units. The number of images at execution time is not determined by the program, but by some compiler-specific method. The number of images is fixed during execution; images cannot be created or destroyed. The intrinsic function NUM_IMAGES() returns the number of images. Each image has an “image index”; this is a positive integer from 1 to the number of images. The intrinsic function THIS_IMAGE() returns the image index of the executing image.

10.2.3 Coarrays

Coarrays are variables that can be directly accessed by another image; they must have the ALLOCATABLE or SAVE attribute or be a dummy argument.

A coarray has a “corank”, which is the number of “codimensions” it has. Each codimension has a lower “cobound” and an upper cobound, determining the “coshape”. The upper cobound of the last codimension is “*”; rather like an assumed-size array. The “cosubscripts” determine the image index of the reference, in the same way that the subscripts of an array determine the array element number. Again, like an assumed-size array, the image index must be less than or equal to the number of images.

A coarray can be a scalar or an array. It cannot have the POINTER attribute, but it can have pointer components.

As well as variables, coarray components are possible. In this case, the component must be an ALLOCATABLE coarray, and any variable with such a component must be a dummy argument or have the SAVE attribute.

10.2.4 Declaring coarrays

A coarray has a coarray-spec which is declared with square brackets after the variable name, or with the CODIMENSION attribute or statement. For example,

    REAL a[100,*]
    REAL,CODIMENSION[-10:10,-10:*] :: b
    CODIMENSION c[*]

declares the coarray A to have corank 2 with lower “cobounds” both 1 and the first upper cobound 100, the coarray B to have corank 2 with lower cobounds both −10 and the first upper cobound 10, and the coarray C to have corank 1 and lower cobound 1. Note that for non-allocatable coarrays, the coarray-spec must always declare the last upper cobound with an asterisk, as this will vary depending on the number of images.

An ALLOCATABLE coarray is declared with a deferred-coshape-spec, for example,

    REAL,ALLOCATABLE :: d[:,:,:,:]

declares the coarray D to have corank 4.

10.2.5 Accessing coarrays on other images

To access another image's copy of a coarray, cosubscripts are used following the coarray name in square brackets; this is called “coindexing”, and such an object is a “coindexed object”. For example, given

    REAL,SAVE :: e[*]

the coindexed object e[1] refers to the copy of E on image 1, and e[13] refers to the copy of E on image 13. For a more complicated example: given

    REAL,SAVE :: f[10,21:30,0:*]

the reference f[3,22,1] refers to the copy of F on image 113. There is no correlation between image numbers and any topology of the computer, so it is probably best to avoid complicated codimensions, especially if different coarrays have different coshape.

When a coarray is an array, you cannot put the cosubscripts directly after the array name, but must use array section notation instead. For example, with

    REAL,SAVE :: g(10,10)[*]

the reference g[inum] is invalid, to refer to the whole array G on image INUM you need to use g(:,:)[inum] instead.

Similarly, to access a single element of G, the cosubscripts follow the subscripts, e.g. g(i,j)[inum].

Finally, note that when a coarray is accessed, whether by its own image or remotely, the segment ordering rules (see next section) must be obeyed. This is to avoid nonsense answers from data races.

10.2.6 Segments and synchronisation

Execution on each image is divided into segments, by “image control statements”. The segments on a single image are ordered: each segment follows the preceding segment. Segments on different images may be ordered (one following the other) by synchronisation, otherwise they are unordered.

If a coarray is defined (assigned a value) in a segment on image I, another image J is only allowed to reference or define it in a segment that follows the segment on I.

The image control statements, and their synchronisation effects, are as follows.

SYNC ALL: synchronises with corresponding SYNC ALL statement executions on other images; the segment following the n^th execution of a SYNC ALL statement on one image follows all the segments that preceded the n^th execution of a SYNC ALL statement on every other image.
SYNC IMAGES (list): synchronises with corresponding SYNC IMAGES statement executions on the images in list, which is an integer expression that may be scalar or a vector. Including the invoking image number in list has no effect. The segment following the n^th execution of a SYNC IMAGES statement on image I with the image number J in its list follows the segments on image J before its n^th execution of SYNC IMAGES with I in its list.
SYNC IMAGES (*): is equivalent to SYNC IMAGES with every image no. in its list, e.g. SYNC IMAGES ([(i,i=1,NUM_IMAGES())]).
SYNC MEMORY: This only acts as a segment divider, without synchronising with any other image. It may be useful for user-defined orderings when some other mechanism has been used to synchronise.
ALLOCATE or DEALLOCATE: with a coarray object being allocated or deallocated. This synchronises all images, which must execute the same ALLOCATE or DEALLOCATE statement.
CRITICAL and END CRITICAL: Only one image can execute a CRITICAL construct at a time. The code inside a CRITICAL construct forms a segment, which follows the previous execution (on whatever image) of the CRITICAL construct.
LOCK and UNLOCK: The segment following a LOCK statements that locks a particular lock variable follows the UNLOCK statement that previously unlocked the variable.
END statement: An END BLOCK, END FUNCTION, or END SUBROUTINE statement that causes automatic deallocation of a local ALLOCATABLE coarray, synchronises with all images (which must execute the same END statement).
MOVE_ALLOC intrinsic: Execution of the intrinsic subroutine MOVE_ALLOC with coarray arguments synchronises all images, which must execute the same CALL statement.

Note that image control statements have side-effects, and therefore are not permitted in pure procedures or within DO CONCURRENT constructs.

10.2.7 Allocating and deallocating coarrays

When you allocate an ALLOCATABLE coarray, you must give the desired cobounds in the ALLOCATE statement. For example,

    REAL,ALLOCATABLE :: x(:,:,:)[:,:]
    ...
    ALLOCATE(x(100,100,3)[1:10,*])

Note that the last upper cobound must be an asterisk, the same as when declaring an explicit-coshape coarray.

When allocating a coarray there is a synchronisation: all images must execute the same ALLOCATE statement, and all the bounds, type parameters, and cobounds of the coarray must be the same on all images.

Similarly, there is a synchronisation when a coarray is deallocated, whether by a DEALLOCATE statement or automatic deallocation by an END statement; every image must execute the same statement.

Note that the usual automatic reallocation of allocatable variables in an intrinsic assignment statement, e.g. when the expression is an array of a different shape, is not available for coarrays. An allocatable coarray variable being assigned to must already be allocated and be conformable with the expression; furthermore, if it has deferred type parameters they must have the same values, and if it is polymorphic it must have the same dynamic type.

10.2.8 Critical constructs

The CRITICAL construct provides a mechanism for ensuring that only one image at a time executes a code segment. For example,

    CRITICAL
      ...do something
    END CRITICAL

If an image I arrives at the CRITICAL statement while another image J is executing the block of the construct, it will wait until image J has executed the END CRITICAL statement before continuing. Thus the CRITICAL — END CRITICAL segment on image I follows the equivalent segment on image J.

As a construct, this may have a name, e.g.

    critsec: CRITICAL
      ...
    END CRITICAL critsec

The name has no effect on the operation of the construct. Each CRITICAL construct is separate from all others, and has no effect on their execution.

10.2.9 Lock variables

A “lock variable” is a variable of the type LOCK_TYPE, defined in the intrinsic module ISO_FORTRAN_ENV. A lock variable must be a coarray, or a component of a coarray. It is initially “unlocked”; it is locked by execution of a LOCK statement, and unlocked by execution of an UNLOCK statement. Apart from those statements, it cannot appear in any variable definition context, other than as the actual argument for an INTENT(INOUT) dummy argument.

Execution of the segment after a LOCK statement successfully locks the variable follows execution of the segment before the UNLOCK statement on the image that unlocked it. For example,

    INTEGER FUNCTION get_sequence_number()
      USE iso_fortran_env
      INTEGER :: number = 0
      TYPE(lock_type) lock[*]
      LOCK(lock[1])
      number = number + 1
      get_sequence_number = number
      UNLOCK(lock[1])
    END FUNCTION

If the variable lock on image 1 is locked when the LOCK statement is executed, it will wait for it to become unlocked before continuing. Thus the function get_sequence_number() provides an one-sided ordering relation: the segment following a call that returned the value N will follow every segment that preceded a call that returned a value less than N.

Conditional locking is provided with the ACQUIRED_LOCK= specifier; if this specifier is present, the executing image only acquires the lock if it was previously unlocked. For example,

    LOGICAL gotit
    LOCK(lock[1],ACQUIRED_LOCK=gotit)
    IF (gotit) THEN
      ! We have the lock.
    ELSE
      ! We do not have the lock - some other image does.
    END IF

It is an error for an image to try to LOCK a variable that is already locked to that image, or to UNLOCK a variable that is already unlocked, or that is locked to another image. If the STAT= specifier is used, these errors will return the values STAT_LOCKED, STAT_UNLOCKED, or STAT_LOCKED_OTHER_IMAGE respectively (these named constants are provided by the intrinsic module ISO_FORTRAN_ENV).

10.2.10 Atomic coarray accessing

As an exception to the segment ordering rules, a coarray that is an integer of kind ATOMIC_INT_KIND or a logical of kind ATOMIC_LOGICAL_KIND (these named constants are provided by the intrinsic module ISO_FORTRAN_ENV), can be defined with the intrinsic subroutine ATOMIC_DEFINE, or referenced by the intrinsic subroutine ATOMIC_REF. For example,

    MODULE stopping
      USE iso_fortran_env
      LOGICAL(atomic_logical_kind),PRIVATE :: stop_flag[*] = .FALSE.
    CONTAINS
      SUBROUTINE make_it_stop
        CALL atomic_define(stop_flag[1],.TRUE._atomic_logical_kind)
      END SUBROUTINE
      LOGICAL FUNCTION please_stop()
        CALL atomic_ref(please_stop,stop_flag[1])
      END FUNCTION
    END MODULE

In this example, it is perfectly valid for any image to call make_it_stop, and for any other image to invoke the function please_stop(), without any regard for segments. (On a distributed memory machine it might take some time for changes to the atomic variable to be visible on other images, but they should eventually get the message.)

Note that ordinary assignment and referencing should not be mixed with calls to the atomic subroutines, as ordinary assignment and referencing are always subject to the segment ordering rules.

10.2.11 Normal termination of execution

If an image executes a STOP statement, or the END PROGRAM statement, normal termination is initiated. The other images continue execution, and all data on the “stopped” image remains; other images can continue to reference and define coarrays on the stopped image.

When normal termination has been initiated on all images, the program terminates.

10.2.12 Error termination

If any image terminates due to an error, for example an input/output error in an input/output statement that does not have any IOSTAT= or ERR= specifier, the entire program is error terminated. On a distributed memory machine it may take some time for the error termination messages to reach every image, so the termination might not be immediate.

The ERROR STOP statement initiates error termination.

10.2.13 Fault tolerance

The Fortran 2018 standard adds many features for detecting, simulating, and recovering from image failure. For example, the FAIL IMAGE statement causes the executing image to fail (stop responding to accesses from other images). These extensions are listed in the detailed syntax below, even though they are not part of the Fortran 2008 standard.

The FAIL IMAGE statement itself is not very useful when the number of images is equal to one, as it inevitably causes complete program failure.

10.2.14 Detailed syntax of coarray features

Coindexed object (data object designator):

In a data object designator, a part (component or base object) that is a coarray can include an image selector: part-name [ ( section-subscript-list ) ] [ image-selector ]

where part-name identifies a coarray, and image-selector is

left-bracket cosubscript-list [, image-selector-spec ] right-bracket

The number of cosubscripts must be equal to the corank of part-name. If image-selector appears and part-name is an array, section-subscript-list must also appear. The optional image-selector-spec is Fortran 2018 (part of the fault tolerance feature), and is a comma-separated list of one or more of the following specifiers:

STAT = scalar-int-variable
TEAM = team-value
TEAM_NUMBER = scalar-int-expression

A team-value must be a scalar expression of type TEAM_TYPE from the intrinsic module ISO_FORTRAN_ENV. The STAT= variable is assigned zero if the reference or definition was successful, and the value STAT_FAILED if the image referenced has failed.

CRITICAL construct:

[ construct-name : ] CRITICAL [ ( [ sync-stat-list ] ) ]
block
END CRITICAL [ construct-name ]

where the optional sync-stat-list is a STAT= specifier, an ERRMSG= specifier, or both (separated by a comma). Note: The optional parentheses and sync-stat-list are Fortran 2018.

The block is not permitted to contain:

a RETURN or STOP statement;
an image control statement;
a branch whose target is outside the construct.

FAIL IMAGE statement:

FAIL IMAGE

Note: This statement is Fortran 2018.

LOCK statement:

LOCK ( lock-variable [, lock-stat-list ] )

where lock-stat-list is a comma-separated list of one or more of the following:

ACQUIRED_LOCK = scalar-logical-variable
ERRMSG = scalar-default-character-variable
STAT = scalar-int-variable

and lock-variable is a scalar variable of type LOCK_TYPE from the intrinsic module ISO_FORTRAN_ENV.

SYNC ALL statement:

SYNC ALL [ ( [ sync-stat-list ] ) ]

SYNC IMAGES statement:

SYNC IMAGES ( image-set [, sync-stat-list ] )

where image-set is an asterisk, or an integer expression that is scalar or of rank one.

SYNC MEMORY statement:

SYNC MEMORY [ ( [ sync-stat-list ] ) ]

UNLOCK statement:

UNLOCK ( lock-variable [, sync-stat-list ] )

Note:

The variables in sync-stat-list or lock-stat-list are not permitted to be coindexed objects, nor may they depend on anything else in the statement.

10.2.15 Intrinsic procedures and coarrays

    SUBROUTINE ATOMIC_DEFINE(ATOM, VALUE, STAT)

ATOM: is INTENT(OUT) scalar INTEGER(ATOMIC_INT_KIND) or LOGICAL(ATOMIC_LOGICAL_KIND), and must be a coarray or a coindexed object.
VALUE: is scalar with the same type as ATOM.
STAT: (Optional) is scalar Integer and must have a decimal exponent range of at least four. It must not be coindexed.

The variable ATOM is atomically assigned the value of VALUE, without regard to the segment rules. If STAT is present, it is assigned a positive value if an error occurs, and zero otherwise. Note: STAT is part of Fortran 2018.

    SUBROUTINE ATOMIC_REF(VALUE, ATOM, STAT)

VALUE: is INTENT(OUT) scalar with the same type as ATOM.
ATOM: is scalar INTEGER(ATOMIC_INT_KIND) or LOGICAL(ATOMIC_LOGICAL_KIND), and must be a coarray or a coindexed object.
STAT: (Optional) is scalar Integer and must have a decimal exponent range of at least four. It must not be coindexed.

The value of ATOM is atomically read, without regard to the segment rules, and then assigned to the variable VALUE. If STAT is present, it is assigned a positive value if an error occurs, and zero otherwise. Note: STAT is part of Fortran 2018.

    INTEGER FUNCTION IMAGE_INDEX(COARRAY, SUB)

COARRAY: a coarray of any type.
SUB: an integer vector whose size is equal to the corank of COARRAY.

If the value of SUB is a valid set of cosubscripts for COARRAY, the value of the result is the image index of the image they will reference, otherwise the result has the value zero. For example, if X is declared with cobounds [11:20,13:*], the result of IMAGE_INDEX(X,[11,13]) will be equal to one, and the result of IMAGE_INDEX(x,[1,1]) will be equal to zero.

    FUNCTION LCOBOUND(COARRAY, DIM , KIND)

COARRAY: coarray of any type and corank N;
DIM: (Optional) scalar Integer in the range 1 to N;
KIND: (Optional) scalar Integer constant expression;
Result: Integer or Integer(Kind=KIND).

If DIM appears, the result is scalar, being the value of the lower cobound of that codimension of COARRAY. If DIM does not appear, the result is a vector of length N containing all the lower cobounds of COARRAY. The actual argument for DIM must not itself be an optional dummy argument.

    SUBROUTINE MOVE_ALLOC(FROM, TO, STAT, ERRMSG)  ! Revised

FROM: an allocatable variable of any type.
TO: an allocatable with the same declared type, type parameters, rank and corank, as FROM.
STAT: INTENT(OUT) scalar Integer with a decimal exponent range of at least four.
ERRMSG: INTENT(INOUT) scalar default character variable.

If FROM and TO are coarrays, the CALL statement is an image control statement that synchronises all images. If STAT is present, it is assigned a positive value if any error occurs, otherwise it is assigned the value zero. If ERRMSG is present and an error occurs, it is assigned an explanatory message. Note: The STAT and ERRMSG arguments are Fortran 2018.

    INTEGER FUNCTION NUM_IMAGES()

This intrinsic function returns the number of images. In this release of the NAG Fortran Compiler, the value will always be equal to one.

    INTEGER FUNCTION THIS_IMAGE()

Returns the image index of the executing image.

    FUNCTION THIS_IMAGE(COARRAY)

Returns an array of type Integer with default kind, with the size equal to the corank of COARRAY, which may be a coarray of any type. The values returned are the cosubscripts for COARRAY that correspond to the executing image.

    INTEGER FUNCTION THIS_IMAGE(COARRAY, DIM)

COARRAY: is a coarray of any type.
DIM: is scalar Integer.

Returns the cosubscript for the codimension DIM that corresponds to the executing image. Note: In Fortran 2008 DIM was not permitted to be an optional dummy argument; Fortran 2018 permits that.

    FUNCTION UCOBOUND(COARRAY, DIM, KIND)

COARRAY: coarray of any type and corank N;
DIM: (Optional) scalar Integer in the range 1 to N;
KIND: (Optional) scalar Integer constant expression;
Result: Integer or Integer(Kind=KIND).

If DIM appears, the result is scalar, being the value of the upper cobound of that codimension of COARRAY. If DIM does not appear, the result is a vector of length N containing all the upper cobounds of COARRAY. The actual argument for DIM must not itself be an optional dummy argument.

Note that if COARRAY has corank N>1, and the number of images in the current execution is not an integer multiple of the coextents up to codimension N−1, the images do not make a full rectangular pattern. In this case, the value of the last upper cobound is the maximum value that a cosubscript can take for that codimension; e.g. with a coarray-spec of [1:3,1:*] and four images in the execution, the last upper cobound will be equal to 2 because the cosubscripts [1,2] are valid even though [2,2] and [2,3] are not.