Given a set of features and a scoring mechanism for any subset of those features, h05abc selects the best subsets of size using a direct communication branch and bound algorithm.
The function may be called by the names: h05abc, nag_mip_best_subset_given_size or nag_best_subset_given_size.
3Description
Given , a set of unique features and a scoring mechanism defined for all then h05abc is designed to find , an optimal subset of size . Here denotes the cardinality of , the number of elements in the set.
The definition of the optimal subset depends on the properties of the scoring mechanism, if
(1)
then the optimal subset is defined as one of the solutions to
else if
(2)
then the optimal subset is defined as one of the solutions to
If neither of these properties hold then h05abc cannot be used.
As well as returning the optimal subset, , h05abc can return the best solutions of size . If
denotes the th best subset, for , then the th best subset is defined as the solution to either
or
depending on the properties of .
The solutions are found using a branch and bound method, where each node of the tree is a subset of . Assuming that (1) holds then a particular node, defined by subset , can be trimmed from the tree if
where
is the th highest score we have observed so far for a subset of size , i.e., our current best guess of the score for the th best subset. In addition, because of (1) we can also drop all nodes defined by any subset where , thus avoiding the need to enumerate the whole tree. Similar short cuts can be taken if (2) holds. A full description of this branch and bound algorithm can be found in Ridout (1988).
Rather than calculate the score at a given node of the tree h05abc utilizes the fast branch and bound algorithm of Somol et al. (2004), and attempts to estimate the score where possible. For each feature, , two values are stored, a count and , an estimate of the contribution of that feature. An initial value of zero is used for both and . At any stage of the algorithm where both
and
have been calculated (as opposed to estimated), the estimated contribution of the feature is updated to
and is incremented by , therefore, at each stage is the mean contribution of observed so far and is the number of observations used to calculate that mean.
As long as , for the user-supplied constant , then rather than calculating this function estimates it using or if has been estimated, where is a user-supplied scaling factor. An estimated score is never used to trim a node or returned as the optimal score.
Setting in this function will cause the algorithm to always calculate the scores, returning to the branch and bound algorithm of Ridout (1988). In most cases it is preferable to use the fast branch and bound algorithm, by setting , unless the score function is iterative in nature, i.e., must have been calculated before can be calculated.
h05abc is a direct communication version of h05aac.
4References
Narendra P M and Fukunaga K (1977) A branch and bound algorithm for feature subset selection IEEE Transactions on Computers9 917–922
Ridout M S (1988) Algorithm AS 233: An improved branch and bound algorithm for feature subset selection Journal of the Royal Statistics Society, Series C (Applied Statistics) (Volume 37)1 139–147
Somol P, Pudil P and Kittler J (2004) Fast branch and bound algorithms for optimal feature selection IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume 26)7 900–912
5Arguments
1: – IntegerInput
On entry: flag indicating whether the scoring function is increasing or decreasing.
, i.e., the subsets with the largest score will be selected.
, i.e., the subsets with the smallest score will be selected.
For all and .
Constraint:
or .
2: – IntegerInput
On entry: , the number of features in the full feature set.
Constraint:
.
3: – IntegerInput
On entry: , the number of features in the subset of interest.
Constraint:
.
4: – IntegerInput
On entry: , the maximum number of best subsets required. The actual number of subsets returned is given by la on final exit. If on final exit then NE_TOO_MANY is returned.
Constraint:
.
5: – Integer *Output
On exit: the number of best subsets returned.
6: – doubleOutput
On exit: holds the score for the la best subsets returned in bz.
7: – IntegerOutput
Note: where appears in this document, it refers to the array element
.
On exit: the th best subset is constructed by dropping the features specified in
, for and , from the set of all features, . The score for the th best subset is given in .
8: – function, supplied by the userExternal Function
On entry: , the number of features in the full feature set.
2: – IntegerInput
On entry: flag indicating whether the intermediate subsets should be constructed by dropping features from the full set () or adding features to the empty set (). See score for additional details.
On entry: , for , contains the list of features which, along with those specified in a, define the subsets whose score is required. See score for additional details.
5: – IntegerInput
On entry: if , the number of subsets for which a score must be returned.
If , the score for a single subset should be returned. See score for additional details.
6: – const IntegerInput
On entry: , for , contains the list of features which, along with those specified in z, define the subsets whose score is required. See score for additional details.
7: – doubleOutput
On exit: the value
, for , the score associated with the th subset. is constructed as follows:
is constructed by dropping the features specified in the first lz elements of z and the single feature given in from the full set of features, . The subset will, therefore, contain features.
is constructed by adding the features specified in the first lz elements of z and the single feature specified in to the empty set, . The subset will, therefore, contain features.
In both cases the individual features are referenced by the integers to m with indicating the first feature, the second, etc., for some arbitrary ordering of the features, chosen by you prior to calling h05abc. For example, might refer to the first variable in a particular set of data, the second, etc..
If , the score for a single subset should be returned. This subset is constructed by adding or removing only those features specified in the first lz elements of z. If , this subset will either be or .
8: – Nag_Comm *
Pointer to structure of type Nag_Comm; the following members are relevant to f.
user – double *
iuser – Integer *
p – Pointer
The type Pointer will be void *. Before calling h05abc you may allocate memory and initialize these pointers with various quantities for use by f when called from h05abc (see Section 3.1.1 in the Introduction to the NAG Library CL Interface).
9: – Integer *Input/Output
On entry: .
On exit: set info to a nonzero value if you wish h05abc to terminate with NE_USER_STOP.
Note:f should not return floating-point NaN (Not a Number) or infinity values, since these are not handled by h05abc. If your code inadvertently does return any NaNs or infinities, h05abc is likely to produce unexpected results.
9: – IntegerInput
On entry: , the minimum number of times the effect of each feature, , must have been observed before is estimated from as opposed to being calculated directly.
If then is never estimated. If then is set to .
10: – doubleInput
On entry: , the scaling factor used when estimating scores. If then is used.
11: – const doubleInput
On entry: a measure of the accuracy of the scoring function, .
Letting , then when confirming whether the scoring function is strictly increasing or decreasing (as described in mincr), or when assessing whether a node defined by subset can be trimmed, then any values in the range are treated as being numerically equivalent.
If then , otherwise .
If then , otherwise .
In most situations setting both and to zero should be sufficient. Using a nonzero value, when one is not required, can significantly increase the number of subsets that need to be evaluated.
12: – Nag_Comm *
The NAG communication argument (see Section 3.1.1 in the Introduction to the NAG Library CL Interface).
13: – NagError *Input/Output
The NAG error argument (see Section 7 in the Introduction to the NAG Library CL Interface).
6Error Indicators and Warnings
NE_ALLOC_FAIL
Dynamic memory allocation failed.
See Section 3.1.2 in the Introduction to the NAG Library CL Interface for further information.
NE_BAD_PARAM
On entry, argument had an illegal value.
NE_INT
On entry, .
Constraint: .
On entry, .
Constraint: or .
On entry, .
Constraint: .
NE_INT_2
On entry, and .
Constraint: .
NE_INTERNAL_ERROR
An internal error has occurred in this function. Check the function call and any array sizes. If the call is correct then please contact NAG for assistance.
See Section 7.5 in the Introduction to the NAG Library CL Interface for further information.
NE_NO_LICENCE
Your licence key may have expired or may not have been installed correctly.
See Section 8 in the Introduction to the NAG Library CL Interface for further information.
NE_REAL
On exit from f, , which is inconsistent with the score for the parent node. Score for the parent node is .
NE_TOO_MANY
On entry, .
But only best subsets could be calculated.
The subsets returned by h05abc are guaranteed to be optimal up to the accuracy of the calculated scores.
8Parallelism and Performance
Background information to multithreading can be found in the Multithreading documentation.
h05abc is threaded by NAG for parallel execution in multithreaded implementations of the NAG Library.
Please consult the X06 Chapter Introduction for information on how to control and interrogate the OpenMP environment used within this function. Please also consult the Users' Note for your implementation for any additional implementation-specific information.
9Further Comments
The maximum number of unique subsets of size from a set of features is . The efficiency of the branch and bound algorithm implemented in h05abc comes from evaluating subsets at internal nodes of the tree, that is subsets with more than features, and where possible trimming branches of the tree based on the scores at these internal nodes as described in Narendra and Fukunaga (1977). Because of this it is possible, in some circumstances, for more than subsets to be evaluated. This will tend to happen when most of the features have a similar effect on the subset score.
If multiple optimal subsets exist with the same score, and nbest is too small to return them all, then the choice of which of these optimal subsets is returned is arbitrary.
10Example
This example finds the three linear regression models, with five variables, that have the smallest residual sums of squares when fitted to a supplied dataset. The data used in this example was simulated.