Feature selection

Machine learning and data mining

Problems
Classification Clustering Regression Anomaly detection Association rules Reinforcement learning Structured prediction Feature learning Online learning Semi-supervised learning Grammar induction
Supervised learning (classification • regression)
Decision trees Ensembles (Bagging, Boosting, Random forest) k-NN Linear regression Naive Bayes Neural networks Logistic regression Perceptron Support vector machine (SVM) Relevance vector machine (RVM)
Clustering
BIRCH Hierarchical k-means Expectation-maximization (EM) DBSCAN OPTICS Mean-shift
Dimensionality reduction
Factor analysis CCA ICA LDA NMF PCA t-SNE
Structured prediction
Graphical models (Bayes net, CRF, HMM)
Anomaly detection
k-NN Local outlier factor
Neural nets
Autoencoder Deep learning Multilayer perceptron RNN Restricted Boltzmann machine SOM Convolutional neural network
Theory
Bias-variance dilemma Computational learning theory Empirical risk minimization PAC learning Statistical learning VC theory
Machine learning portal Computer science portal Statistics portal

In machine learning and statistics, feature selection, also known as variable selection, attribute selection or variable subset selection, is the process of selecting a subset of relevant features for use in model construction. The central assumption when using a feature selection technique is that the data contains many redundant or irrelevant features. Redundant features are those which provide no more information than the currently selected features, and irrelevant features provide no useful information in any context.

Feature selection techniques is to be distinguished from feature extraction. Feature extraction creates new features from functions of the original features, whereas feature selection returns a subset of the features. Feature selection techniques are often used in domains where there are many features and comparatively few samples (or data points). The archetypal case is the use of feature selection in analysing DNA microarrays, where there are many thousands of features, and a few tens to hundreds of samples. Feature selection techniques provide three main benefits when constructing predictive models:

improved model interpretability,
shorter training times,
enhanced generalisation by reducing overfitting.

Feature selection is also useful as part of the data analysis process, as it shows which features are important for prediction, and how these features are related.

Introduction

A feature selection algorithm can be seen as the combination of a search technique for proposing new feature subsets, along with an evaluation measure which scores the different feature subsets. The simplest algorithm is to test each possible subset of features finding the one which minimises the error rate. This is an exhaustive search of the space, and is computationally intractable for all but the smallest of feature sets. The choice of evaluation metric heavily influences the algorithm, and it is these evaluation metrics which distinguish between the three main categories of feature selection algorithms: wrappers, filters and embedded methods.^[1]

Wrapper methods use a predictive model to score feature subsets. Each new subset is used to train a model, which is tested on a hold-out set. Counting the number of mistakes made on that hold-out set (the error rate of the model) gives the score for that subset. As wrapper methods train a new model for each subset, they are very computationally intensive, but usually provide the best performing feature set for that particular type of model.

Filter methods use a proxy measure instead of the error rate to score a feature subset. This measure is chosen to be fast to compute, whilst still capturing the usefulness of the feature set. Common measures include the mutual information,^[1] the pointwise mutual information,^[2] Pearson product-moment correlation coefficient, inter/intra class distance or the scores of significance tests for each class/feature combinations.^[2]^[3] Filters are usually less computationally intensive than wrappers, but they produce a feature set which is not tuned to a specific type of predictive model. This lack of tuning means a feature set from a filter is more general than the set from a wrapper, usually giving lower prediction performance than a wrapper. However the feature set doesn't contain the assumptions of a prediction model, and so is more useful for exposing the relationships between the features. Many filters provide a feature ranking rather than an explicit best feature subset, and the cut off point in the ranking is chosen via cross-validation. Filter methods have also been used as a preprocessing step for wrapper methods, allowing a wrapper to be used on larger problems.

Embedded methods are a catch-all group of techniques which perform feature selection as part of the model construction process. The exemplar of this approach is the LASSO method for constructing a linear model, which penalises the regression coefficients, shrinking many of them to zero. Any features which have non-zero regression coefficients are 'selected' by the LASSO algorithm. Improvements to the LASSO include Bolasso which bootstraps samples,^[4] and FeaLect which scores all the features based on combinatorial analysis of regression coefficients.^[5] One other popular approach is the Recursive Feature Elimination algorithm, commonly used with Support Vector Machines to repeatedly construct a model and remove features with low weights. These approaches tend to be between filters and wrappers in terms of computational complexity.

In statistics, the most popular form of feature selection is stepwise regression, which is a wrapper technique. It is a greedy algorithm that adds the best feature (or deletes the worst feature) at each round. The main control issue is deciding when to stop the algorithm. In machine learning, this is typically done by cross-validation. In statistics, some criteria are optimized. This leads to the inherent problem of nesting. More robust methods have been explored, such as branch and bound and piecewise linear network.

Subset selection

Subset selection evaluates a subset of features as a group for suitability. Subset selection algorithms can be broken up into Wrappers, Filters and Embedded. Wrappers use a search algorithm to search through the space of possible features and evaluate each subset by running a model on the subset. Wrappers can be computationally expensive and have a risk of over fitting to the model. Filters are similar to Wrappers in the search approach, but instead of evaluating against a model, a simpler filter is evaluated. Embedded techniques are embedded in and specific to a model.

Many popular search approaches use greedy hill climbing, which iteratively evaluates a candidate subset of features, then modifies the subset and evaluates if the new subset is an improvement over the old. Evaluation of the subsets requires a scoring metric that grades a subset of features. Exhaustive search is generally impractical, so at some implementor (or operator) defined stopping point, the subset of features with the highest score discovered up to that point is selected as the satisfactory feature subset. The stopping criterion varies by algorithm; possible criteria include: a subset score exceeds a threshold, a program's maximum allowed run time has been surpassed, etc.

Alternative search-based techniques are based on targeted projection pursuit which finds low-dimensional projections of the data that score highly: the features that have the largest projections in the lower-dimensional space are then selected.

Search approaches include:

Exhaustive
Best first
Simulated annealing
Genetic algorithm
Greedy forward selection
Greedy backward elimination
Targeted projection pursuit
Scatter Search^[6]
Variable Neighborhood Search^[7]

Two popular filter metrics for classification problems are correlation and mutual information, although neither are true metrics or 'distance measures' in the mathematical sense, since they fail to obey the triangle inequality and thus do not compute any actual 'distance' – they should rather be regarded as 'scores'. These scores are computed between a candidate feature (or set of features) and the desired output category. There are, however, true metrics that are a simple function of the mutual information;^[8] see here.

Other available filter metrics include:

Class separability
- Error probability
- Inter-class distance
- Probabilistic distance
- Entropy
Consistency-based feature selection
Correlation-based feature selection

Optimality criteria

The choice of optimality criteria is difficult as there are multiple objectives in a feature selection task. Many common ones incorporate a measure of accuracy, penalised by the number of features selected (e.g. the Bayesian information criterion). The oldest are Mallows's C_p statistic and Akaike information criterion (AIC). These add variables if the t-statistic is bigger than $\sqrt{2}$ .

Other criteria are Bayesian information criterion (BIC) which uses $\sqrt{\log{n}}$ , minimum description length (MDL) which asymptotically uses $\sqrt{\log{n}}$ , Bonnferroni / RIC which use $\sqrt{2\log{p}}$ , maximum dependency feature selection, and a variety of new criteria that are motivated by false discovery rate (FDR) which use something close to $\sqrt{2\log{\frac{p}{q}}}$ .

Structure Learning

Filter feature selection is a specific case of a more general paradigm called Structure Learning. Feature selection finds the relevant feature set for a specific target variable whereas structure learning finds the relationships between all the variables, usually by expressing these relationships as a graph. The most common structure learning algorithms assume the data is generated by a Bayesian Network, and so the structure is a directed graphical model. The optimal solution to the filter feature selection problem is the Markov blanket of the target node, and in a Bayesian Network, there is a unique Markov Blanket for each node.^[9]

Minimum-redundancy-maximum-relevance (mRMR) feature selection

Peng et al.^[10] proposed a feature selection method that can use either mutual information, correlation, or distance/similarity scores to select features. The aim is to penalise a feature's relevancy by its redundancy in the presence of the other selected features. The relevance of a feature set S for the class c is defined by the average value of all mutual information values between the individual feature f_i and the class c as follows:

$D(S,c) = \frac{1}{|S|}\sum_{f_{i}\in S}I(f_{i};c)$ .

The redundancy of all features in the set S is the average value of all mutual information values between the feature f_i and the feature f_j:

$R(S) = \frac{1}{|S|^{2}}\sum_{f_{i},f_{j}\in S}I(f_{i};f_{j})$

The mRMR criterion is a combination of two measures given above and is defined as follows:

$\mathrm{mRMR}= \max_{S} \left[\frac{1}{|S|}\sum_{f_{i}\in S}I(f_{i};c) - \frac{1}{|S|^{2}}\sum_{f_{i},f_{j}\in S}I(f_{i};f_{j})\right].$

Suppose that there are n full-set features. Let x_i be the set membership indicator function for feature f_i, so that x_i=1 indicates presence and x_i=0 indicates absence of the feature f_i in the globally optimal feature set. Let c_i=I(f_i;c) and a_ij=I(f_i;f_j). The above may then be written as an optimization problem:

$\mathrm{mRMR}= \max_{x\in \{0,1\}^{n}} \left[\frac{\sum^{n}_{i=1}c_{i}x_{i}}{\sum^{n}_{i=1}x_{i}} - \frac{\sum^{n}_{i,j=1}a_{ij}x_{i}x_{j}} {(\sum^{n}_{i=1}x_{i})^{2}}\right].$

The mRMR algorithm is an approximation of the theoretically optimal maximum-dependency feature selection algorithm that maximizes the mutual information between the joint distribution of the selected features and the classification variable. As mRMR approximates the combinatorial estimation problem with a series of much smaller problems, each of which only involves two variables, it thus uses pairwise joint probabilities which are more robust. In certain situations the algorithm may underestimate the usefulness of features as it has no way to measure interactions between features which can increase relevancy. This can lead to poor performance^[11] when the features are individually useless, but are useful when combined (a pathological case is found when the class is a parity function of the features). Overall the algorithm is more efficient (in terms of the amount of data required) than the theoretically optimal max-dependency selection, yet produces a feature set with little pairwise redundancy.

mRMR is an instance of a large class of filter methods which trade off between relevancy and redundancy in different ways.^[11]^[12]

Global optimization formulations

mRMR is a typical example of an incremental greedy strategy for feature selection: once a feature has been selected, it cannot be deselected at a later stage. While mRMR could be optimized using floating search to reduce some features, it might also be reformulated as a global quadratic programing optimization problem as follows:^[13]

$\mathrm{QPFS}: \min_\mathbf{x} \left\{ \alpha \mathbf{x}^T H \mathbf{x} - \mathbf{x}^T F\right\} \quad \mbox{s.t.} \ \sum_{i=1}^n x_i=1, x_i\geq 0$

where $F_{n\times1}=[I(f_1;c),\ldots, I(f_n;c)]^T$ is the vector of feature relevancy assuming there are $n$ features in total, $H_{n\times n}=[I(f_i;f_j)]_{i,j=1\ldots n}$ is the matrix of feature pairwise redundancy, and $\mathbf{x}_{n\times 1}$ represents relative feature weights. QPFS is solved via quadratic programming. It is recently shown that QFPS is biased towards features with smaller entropy,^[14] due to its placement of the feature self redundancy term $I(f_i;f_i)$ on the diagonal of $H$ .

Another global formulation for the mutual information based feature selection problem is based on the conditional relevancy:^[14]

$\mathrm{SPEC_{CMI}}: \max_{\mathbf{x}} \left\{\mathbf{x}^T Q \mathbf{x}\right\} \quad \mbox{s.t.}\ \|\mathbf{x}\|=1, x_i\geq 0$

where $Q_{ii}=I(f_i;c)$ and $Q_{ij}=I(f_i;c|f_j), i\ne j$ .

An advantage of $\mathrm{SPEC_{CMI}}$ is that it can be solved simply via finding the dominant eigenvector of $Q$ , thus is very scalable. $\mathrm{SPEC_{CMI}}$ also handles second-order feature interaction.

Correlation feature selection

The Correlation Feature Selection (CFS) measure evaluates subsets of features on the basis of the following hypothesis: "Good feature subsets contain features highly correlated with the classification, yet uncorrelated to each other".^[15]^[16] The following equation gives the merit of a feature subset S consisting of k features:

$Merit_{S_{k}} = \frac{k\overline{r_{cf}}}{\sqrt{k+k(k-1)\overline{r_{ff}}}}.$

Here, $\overline{r_{cf}}$ is the average value of all feature-classification correlations, and $\overline{r_{ff}}$ is the average value of all feature-feature correlations. The CFS criterion is defined as follows:

$\mathrm{CFS} = \max_{S_k} \left[\frac{r_{c f_1}+r_{c f_2}+\cdots+r_{c f_k}} {\sqrt{k+2(r_{f_1 f_2}+\cdots+r_{f_i f_j}+ \cdots + r_{f_k f_1 })}}\right].$

The $r_{cf_{i}}$ and $r_{f_{i}f_{j}}$ variables are referred to as correlations, but are not necessarily Pearson's correlation coefficient or Spearman's ρ. Dr. Mark Hall's dissertation uses neither of these, but uses three different measures of relatedness, minimum description length (MDL), symmetrical uncertainty, and relief.

Let x_i be the set membership indicator function for feature f_i; then the above can be rewritten as an optimization problem:

$\mathrm{CFS} = \max_{x\in \{0,1\}^{n}} \left[\frac{(\sum^{n}_{i=1}a_{i}x_{i})^{2}} {\sum^{n}_{i=1}x_i + \sum_{i\neq j} 2b_{ij} x_i x_j }\right].$

The combinatorial problems above are, in fact, mixed 0–1 linear programming problems that can be solved by using branch-and-bound algorithms.^[17]

Regularized trees

The features from a decision tree or a tree ensemble are shown to be redundant. A recent method called regularized tree^[18] can be used for feature subset selection. Regularized trees penalize using a variable similar to the variables selected at previous tree nodes for splitting the current node. Regularized trees only need build one tree model (or one tree ensemble model) and thus are computationally efficient.

Regularized trees naturally handle numerical and categorical features, interactions and nonlinearities. They are invariant to attribute scales (units) and insensitive to outliers, and thus, require little data preprocessing such as normalization. Regularized random forest (RRF) (RRF) is one type of regularized trees. The guided RRF is an enhanced RRF which is guided by the importance scores from an ordinary random forest.

Overview on metaheuristics methods

A metaheuristic is a general description of an algorithm dedicated to solve difficult (typically NP-hard problem) optimisation problems for which there is no classical solving methods. Generally, a metaheuristic is a stochastics algorithm tending to reach a global optima. There are many metaheuristics, from a simple local search to a complex global search algorithm.

Main principles

The feature selection methods are typically presented in three classes based on how they combine the selection algorithm and the model building

Filter Method

Filter type methods select variables regardless of the model. They are based only on general features like the correlation with the variable to predict. Filter methods suppress the least interesting variables. The others variables will be part of a the model classification, a regression used to classify or a data prediction. These methods are particularly effective in computation time and robust to overfitting.^[19]

However, filter methods tend to select redundant variables because they does not consider the relationships between variables. Therefore, they are mainly used as a pre-process method.

Filter Method for feature selection

Wrapper Method

Methods wrapper evaluate subsets of variables which allows, unlike filter approaches, to detect the possible interactions between variables.^[20] The two main disadvantages of these methods are :

The increasing overfitting risk when the number of observations is insufficient.
The significant computation time when the number of variables is large.

Wrapper Method for Feature selection

Embedded Method

Recently, embedded methods have been proposed to reduce the classification of learning. They try to combine the advantages of both previous methods. The learning algorithm takes advantage of its own variable selection algorithm. So, it needs to know preliminary what a good selection is, which limits their exploitation.^[21]

Embedded method for Feature selection

Application of feature selection metaheuristics

This is a survey of the application of feature selection metaheuristics lately used in the literature. This survey was realized by J. Hammon in her thesis.^[19]

Application	Algorithm	Approach	classifier	Evaluation Function	Ref
SNPs	Feature Selection Feature Similarity	Filter		r²	Phuong 2005 ^[20]
SNPs	Genetic Algorithm	Wrapper	Decision Tree	Classification accuracy (10-fold)	Shah 2004 ^[22]
SNPs	HillClimbing	Filter + Wrapper	Naive Bayesian	Predicted residual sum of squares	Long 2007 ^[23]
SNPs	Simuleating Annealing		Naive bayesian	Classification accuracy (5-fols)	Ustunkar 2011 ^[24]
Segments parole	Ants colony	Wrapper	Artificial Neural Network	MSE	Al-ani 2005 ^[25]
Marketing	Simulated Annealing	Wrapper	Regression	AIC, r²	Meiri 2006 ^[26]
Economy	Simulated Annealing, Genetic Algorithm	Wrapper	Regression	BIC	Kapetanios 2005 ^[27]
Spectral Mass	Genectic Algorithm	Wrapper	Multiple Regression Linear, Partial Least Square	root-mean-square error of prediction	Broadhurst 2007 ^[28]
Microarray	Tabu Search + PSO	Wrapper	Support Vector Machine, K Nearest Neighbor	Euclidian Distance	Chuang 2009 ^[29]
Microarray	PSO + Genetic Algorithm	Wrapper	Support Vector Machine	Classification accuracy (10-fold)	Alba 2007 ^[30]
Microarray	Genetic Algorithm + Iterated Local Search	Embedded	Support Vector Machine	Classification accuracy (10-fold)	Duval 2009 ^[21]
Microarray	Iterated Local Search	Wrapper	Regression	Posterior Probability	Hans 2007 ^[31]
Microarray	Genetic Algorithm	Wrapper	K Nearest Neighbor	Classification accuracy (Leave-one-out cross-validation)	Jirapech-Umpai 2005 ^[32]
Microarray	Hybrid Genetic Algorithm	Wrapper	K Nearest Neighbor	Classification accuracy (Leave-one-out cross-validation)	Oh 2004 ^[33]
Microarray	Genetic Algorithm	Wrapper	Support Vector Machine	Sensibility Specificity	Xuan 2011 ^[34]
Microarray	Genetic Algorithm	Wrapper	All paired, Support Vector Machine	Classification accuracy (Leave-one-out cross-validation)	Peng 2003 ^[35]
Microarray	Genetic Algorithm	Embedded	Support Vector Machine	Classification accuracy (10-fold)	Hernandez 2007 ^[36]
Microarray	Genetic Algorithm	Hybrid	Support Vector Machine	Classification accuracy (Leave-one-out cross-validation)	Huerta 2006 ^[37]
Microarray	Genetic Algorithm		Support Vector Machine	Classification accuracy (10-fold)	Muni 2006 ^[38]
Microarray	Genetic Algorithm	Wrapper	Support Vector Machine	EH-DIALL, CLUMP	Jourdan 2004 ^[39]