Feature Selection

However, there are some heuristic approaches that are often useful. We will look at the following approaches:

Stepwise Selection

A common suggestion for avoiding the consideration of all subsets is to use stepwise selection. There are two standard approaches:
  • Forward selection. Begin by finding the best single feature, and commit to it. In general, given a set of selected features, add the feature that improves performance most.

  • Backward elimination. From a set of remaining features, repeatedly delete the feature that reduces performance the least.