nag_approx_quantiles_arbitrary (g01apc) (PDF version)
g01 Chapter Contents
g01 Chapter Introduction
NAG Library Manual

NAG Library Function Document

nag_approx_quantiles_arbitrary (g01apc)

+ Contents

    1  Purpose
    7  Accuracy

1  Purpose

nag_approx_quantiles_arbitrary (g01apc) finds approximate quantiles from a large arbitrary-sized data stream using an out-of-core algorithm.

2  Specification

#include <nag.h>
#include <nagg01.h>
void  nag_approx_quantiles_arbitrary (Integer *ind, const double rv[], Integer nb, double eps, Integer *np, const double q[], double qv[], Integer nq, double rcomm[], Integer lrcomm, Integer icomm[], Integer licomm, NagError *fail)

3  Description

A quantile is a value which divides a frequency distribution such that there is a given proportion of data values below the quantile. For example, the median of a dataset is the 0.5 quantile because half the values are less than or equal to it.
nag_approx_quantiles_arbitrary (g01apc) uses a slightly modified version of an algorithm described in a paper by Zhang and Wang (2007) to determine ε-approximate quantiles of a large arbitrary-sized data stream of real values, where ε is a user-defined approximation factor. Let m denote the number of data elements processed so far then, given any quantile q0.0,1.0, an ε-approximate quantile is defined as an element in the data stream whose rank falls within q-εm,q+εm . In case of more than one ε-approximate quantile being available, the one closest to qm is used.

4  References

Zhang Q and Wang W (2007) A fast algorithm for approximate quantiles in high speed data streams Proceedings of the 19th International Conference on Scientific and Statistical Database Management IEEE Computer Society 29

5  Arguments

1:     indInteger *Input/Output
On initial entry: must be set to 0.
On entry: indicates the action required in the current call to nag_approx_quantiles_arbitrary (g01apc).
ind=0
Initialize the communication arrays and attempt to process the first nb values from the data stream. eps, rv and nb must be set and licomm must be at least 10.
ind=1
Attempt to process the next block of nb values from the data stream. The calling program must update rv and (if required) nb, and re-enter nag_approx_quantiles_arbitrary (g01apc) with all other parameters unchanged.
ind=2
Continue calculation following the reallocation of either or both of the communication arrays rcomm and icomm.
ind=3
Calculate the nq ε-approximate quantiles specified in q. The calling program must set q and nq and re-enter nag_approx_quantiles_arbitrary (g01apc) with all other parameters unchanged. This option can be chosen only when npexp1.0/eps.
On exit: indicates output from the call.
ind=1
nag_approx_quantiles_arbitrary (g01apc) has processed np data points and expects to be called again with additional data.
ind=2
Either one or more of the communication arrays rcomm and icomm is too small. The new minimum lengths of rcomm and icomm have been returned in icomm[0] and icomm[1] respectively. If the new minimum length is greater than the current length then the corresponding communication array needs to be reallocated, its contents preserved and nag_approx_quantiles_arbitrary (g01apc) called again with all other parameters unchanged.
If there is more data to be processed, it is recommended that lrcomm and licomm are made significantly bigger than the minimum to limit the number of reallocations.
ind=3
nag_approx_quantiles_arbitrary (g01apc) has returned the requested ε-approximate quantiles in qv. These quantiles are based on np data points.
Constraint: ind=0, 1, 2 or 3.
2:     rv[dim]const doubleInput
Note: the dimension, dim, of the array rv must be at least nb when ind=0, 1 or 2.
On entry: if ind=0, 1 or 2, the vector containing the current block of data, otherwise rv is not referenced.
3:     nbIntegerInput
On entry: if ind=0, 1 or 2, the size of the current block of data. The size of blocks of data in array rv can vary; therefore nb can change between calls to nag_approx_quantiles_arbitrary (g01apc).
Constraint: if ind=0, 1 or 2, nb>0.
4:     epsdoubleInput
On entry: approximation factor ε.
Constraint: eps>0.0 ​ and ​eps1.0.
5:     npInteger *Output
On exit: m, the number of elements processed so far.
6:     q[dim]const doubleInput
Note: the dimension, dim, of the array q must be at least nq when ind=3.
On entry: if ind=3, the quantiles to be calculated, otherwise q is not referenced. Note that q[i]=0.0, corresponds to the minimum value and q[i]=1.0 to the maximum value.
Constraint: if ind=3, 0.0q[i-1]1.0, for i=1,2,,nq.
7:     qv[dim]doubleOutput
Note: the dimension, dim, of the array qv must be at least nq when ind=3.
On exit: if ind=3, qv[i] contains the ε-approximate quantiles specified by the value provided in q[i].
8:     nqIntegerInput
On entry: if ind=3, the number of quantiles requested, otherwise nq is not referenced.
Constraint: if ind=3, nq>0.
9:     rcomm[lrcomm]doubleCommunication Array
On entry: if ind=1 or 2 then the first l elements of rcomm as supplied to nag_approx_quantiles_arbitrary (g01apc) must be identical to the first l elements of rcomm returned from the last call to nag_approx_quantiles_arbitrary (g01apc), where l is the value of lrcomm used in the last call. In other words, the contents of rcomm must not be altered between calls to this function. If rcomm needs to be reallocated then its contents must be preserved. If ind=0 then rcomm need not be set.
On exit: rcomm holds information required by subsequent calls to nag_approx_quantiles_arbitrary (g01apc)
10:   lrcommIntegerInput
On entry: the dimension of the array rcomm.
Constraints:
  • if ind=0, lrcomm1;
  • otherwise lrcommicomm[0].
11:   icomm[licomm]IntegerCommunication Array
On entry: if ind=1 or 2 then the first l elements of icomm as supplied to nag_approx_quantiles_arbitrary (g01apc) must be identical to the first l elements of icomm returned from the last call to nag_approx_quantiles_arbitrary (g01apc), where l is the value of licomm used in the last call. In other words, the contents of icomm must not be altered between calls to this function. If icomm needs to be reallocated then its contents must be preserved. If ind=0 then icomm need not be set.
On exit: icomm[0] holds the minimum required length for rcomm and icomm[1] holds the minimum required length for icomm. The remaining elements of icomm are used for communication between subsequent calls to nag_approx_quantiles_arbitrary (g01apc).
12:   licommIntegerInput
On entry: the dimension of the array icomm.
Constraints:
  • if ind=0, licomm10;
  • otherwise licommicomm[1].
13:   failNagError *Input/Output
The NAG error argument (see Section 3.6 in the Essential Introduction).

6  Error Indicators and Warnings

NE_ALLOC_FAIL
Dynamic memory allocation failed.
NE_ARRAY_SIZE
On entry, licomm=value.
Constraint: licomm10.
On entry, lrcomm=value.
Constraint: lrcomm1.
NE_BAD_PARAM
On entry, argument value had an illegal value.
NE_ILLEGAL_COMM
The contents of icomm have been altered between calls to this function.
The contents of rcomm have been altered between calls to this function.
NE_INT
On entry, ind=0, 1 or 2 and nb=value.
Constraint: if ind=0, 1 or 2 then nb>0.
On entry, ind=3 and nq=value.
Constraint: if ind=3 then nq>0.
On entry, ind=value.
Constraint: ind=0, 1, 2 or 3.
NE_INTERNAL_ERROR
An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact NAG for assistance.
NE_Q_OUT_OF_RANGE
On entry, ind=3 and q[value]=value.
Constraint: if ind=3 then 0.0q[i]1.0 for all i.
NE_REAL
On entry, eps=value.
Constraint: 0.0<eps1.0.
NE_TOO_SMALL
Number of data elements streamed, value is not sufficient for a quantile query when eps=value.
Supply more data or reprocess the data with a higher eps value.

7  Accuracy

Not applicable.

8  Parallelism and Performance

nag_approx_quantiles_arbitrary (g01apc) is threaded by NAG for parallel execution in multithreaded implementations of the NAG Library.
Please consult the Users' Note for your implementation for any additional implementation-specific information.

9  Further Comments

The average time taken by nag_approx_quantiles_arbitrary (g01apc) scales as np log 1/ε logεnp .
It is not possible to determine in advance the final size of the communication arrays rcomm and icomm without knowing the size of the dataset. However, if a rough size (n) is known, the speed of the computation can be increased if the sizes of the communication arrays are not smaller than
lrcomm = log2 n×eps+1.0 - 2 × 1.0/eps +1+x+ 2× minx, x/2.0 +1 × y +1 licomm = log2 n×eps+1.0 - 2 × 2 × 1.0/eps +1 + 1 + 2 × x+2× minx, x/2.0 +1 × y + y + 11
where
x= max1, log eps×n / eps y = log2n/x+1.0 +1 .

10  Example

This example computes a list of ε-approximate quantiles. The data is processed in blocks of 20 observations at a time to simulate a situation in which the data is made available in a piecemeal fashion.

10.1  Program Text

Program Text (g01apce.c)

10.2  Program Data

Program Data (g01apce.d)

10.3  Program Results

Program Results (g01apce.r)


nag_approx_quantiles_arbitrary (g01apc) (PDF version)
g01 Chapter Contents
g01 Chapter Introduction
NAG Library Manual

© The Numerical Algorithms Group Ltd, Oxford, UK. 2014