Random subspace method

In machine learning the random subspace method,[1] also called attribute bagging[2] or feature bagging, is an ensemble learning method that attempts to reduce the correlation between estimators in an ensemble by training them on random samples of features instead of the entire feature set.

Motivation

In ensemble learning one tries to combine the models produced by several learners into an ensemble that performs better than the original learners. One way of combining learners is bootstrap aggregating or bagging, which shows each learner a randomly sampled subset of the training points so that the learners will produce different models that can be sensibly averaged.[lower-alpha 1] In bagging, one samples training points with replacement from the full training set.

The random subspace method is similar to bagging except that the features ("attributes", "predictors", "independent variables") are randomly sampled, with replacement, for each learner. Informally, this causes individual learners to not over-focus on features that appear highly predictive/descriptive in the training set, but fail to be as predictive for points outside that set. For this reason, random subspaces are an attractive choice for problems where the number of features is much larger than the number of training points, such as learning from fMRI data[3] or gene expression data.[4]

The random subspace method has been used for decision trees; when combined with "ordinary" bagging of decision trees, the resulting models are called random forests.[5] It has also been applied to linear classifiers,[6] support vector machines,[7] nearest neighbours[8] [9] and other types of classifiers. This method is also applicable to one-class classifiers.[10][11] Recently, the random subspace method has been used in a portfolio selection[12] problem showing its superiority to the conventional resampled portfolio essentially based on Bagging.

Algorithm

An ensemble of models employing the random subspace method can be constructed using the following algorithm:

  1. Let the number of training points be N and the number of features in the training data be D.
  2. Choose L to be the number of individual models in the ensemble.
  3. For each individual model l, choose dl (dl < D) to be the number of input variables for l. It is common to have only one value of dl for all the individual models.
  4. For each individual model l, create a training set by choosing dl features from D with replacement and train the model.

Now, to apply the ensemble model to an unseen point, combine the outputs of the L individual models by majority voting or by combining the posterior probabilities.

Footnotes

  1. ↑ If each learner follows the same, deterministic, algorithm, the models produced are necessarily all the same.

References

  1. ↑ Ho, Tin Kam (1998). "The Random Subspace Method for Constructing Decision Forests" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 20 (8): 832–844. doi:10.1109/34.709601.
  2. ↑ Bryll, R. (2003). "Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets". Pattern Recognition. 36 (6): 1291–1302. doi:10.1016/s0031-3203(02)00121-8.
  3. ↑ Kuncheva, Ludmila; et al. (2010). "Random Subspace Ensembles for fMRI Classification" (PDF). IEEE Transactions on Medical Imaging. 29 (2): 531–542. doi:10.1109/TMI.2009.2037756.
  4. ↑ Bertoni, Alberto; Folgieri, Raffaella; Valentini, Giorgio (2005). "Bio-molecular cancer prediction with random subspace ensembles of support vector machines". Neurocomputing. 63: 535–539. doi:10.1016/j.neucom.2004.07.007.
  5. ↑ Ho, Tin Kam (1995). Random Decision Forest (PDF). Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, 14–16 August 1995. pp. 278–282.
  6. ↑ Skurichina, Marina (2002). "Bagging, boosting and the random subspace method for linear classifiers". Pattern Analysis and Applications. 5 (2): 121–135. doi:10.1007/s100440200011.
  7. ↑ Tao, D. (2006). "Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval". IEEE Transactions on Pattern Analysis and Machine Intelligence. doi:10.1109/tpami.2006.134.
  8. ↑ Ho, Tin Kam (1998). "Nearest Neighbors in Random Subspaces". Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR): 640–648.
  9. ↑ Tremblay, G. (2004). "Optimizing Nearest Neighbour in Random Subspaces using a Multi-Objective Genetic Algorithm" (PDF). 17th International Conference on Pattern Recognition: 208–211.
  10. ↑ Nanni, L. (2006). "Experimental comparison of one-class classifiers for online signature verification". Neurocomputing. 69 (7).
  11. ↑ Cheplygina, Veronika; Tax, David M. J. (2011-06-15). Sansone, Carlo; Kittler, Josef; Roli, Fabio, eds. Multiple Classifier Systems. Lecture Notes in Computer Science. Springer Berlin Heidelberg. pp. 96–105. ISBN 9783642215568. doi:10.1007/978-3-642-21557-5_12.
  12. ↑ Shen, Weiwei; Wang, Jun (2017), "Portfolio Selection via Subset Resampling", Proceedings of AAAI Conference on Artificial Intelligence (AAAI2017)
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.