Feature Selection

However, there are some heuristic approaches that are often useful. We will look at the following approaches:

Stepwise Selection

A common suggestion for avoiding the consideration of all subsets is to use stepwise selection. There are two standard approaches:
  • Forward selection. Begin by finding the best single feature, and commit to it. In general, given a set of selected features, add the feature that improves performance most.

  • Backward elimination. From a set of remaining features, repeatedly delete the feature that reduces performance the least.


    Reference:
    https://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/FS/FS_home.htm
    https://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/FS/stepwise.htm