Vector feature selection

Mutual-information based feature selection considering feature sets rather than single-dimensional features.

The mutual information (MI) among two random variables X and Y, I(X;Y) can be computed from their joint and marginal probability density functions (pdf) fxy, fx and fy. The MI can be expanded to random vectors X and Y, however their pdf estimation becomes much harder.

Take the three types of feature selection methods:

• Forward selection
• Backward elimination
• Exhaustive search

Mutual information based selection methods in the literature traditionally follow the Forward Selection approach by using a variety of scores such as:

• MIM
• MRMR
• JMIM
• etc.

Which consist of different combinations of low-dim MI among the different candidate features and the target, keeping to the trivariate case MI(X,Y;Z) at most and avoiding the hindrance of estimating high-dimensional probability densities.

This repository provides implementations for:

• Mutual Information MI(X;Y)
• Forward selection methods in the literature
• Backward elimination from the methods in the literature
• Exhaustive selection based on the raw MI(X;Y)
• Vectorized versions of MRMR and DISR methods by replacing f(MI(Xi;Y)) -> MI(X;Y)

💥 See next section Shorts for a super brief tutorial on how to use the library.

💥 See subdirectory scripts for more detailed examples.

Shorts

from vfs import *
from vfs.shorthands import df_iris, MRMR
df, features, targets = df_iris()

Mutual information

# Mutual information between two variables
mi = mi_frame(df)(['F1'], ['F2'])
print(mi)

# Mutual information between two groups of variables (vectors)
mi = mi_frame(df)(['F1','F2'], ['F3','F4', 'F5'])
print(mi)

Traditional feature selection

# Select the two best features according to MRMR (forward), using shorthand
summary, __, __ = MRMR(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2)
print(summary)

# Select the best two features according to JMIM (backward), using default func
__, sel, disc = backward_eliminator(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, loss=jmim, mi_fun=mi_frame(df))
print(sel)
print(disc)

Vectorial feature selection

# Select the best three features by testing all feat combinations
sel, score = exhaustive_searcher(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, mi_fun=mi_frame(df))
print(sel)

# Select the best feature vector between two candidates
mifun = mi_frame(df)
aa = ['F1', 'F2']
bb = ['F3', 'F4']
best = aa if mifun(aa, ['F5']) > mifun(bb, ['F5']) else bb

Bibliography

• Battiti, Roberto. 1994. “Using Mutual Information for Selecting Features in Supervised Neural Net Learning.” IEEE Transactions on Neural Networks 5 (4): 537–50.
  
  
• Yang, H, and John Moody. 1999. “Feature Selection Based on Joint Mutual Information.” In Proceedings of International ICSC Symposium on Advances in Intelligent Data Analysis, 1999:22–25. Citeseer.
  
  
• Fleuret, François. 2004. “Fast Binary Feature Selection with Conditional Mutual Information.” Journal of Machine Learning Research 5 (9).
  
  
• Peng, Hanchuan, Fuhui Long, and Chris Ding. 2005. “Feature Selection Based on Mutual Information Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy.” IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (8): 1226–38.
  
  
• Meyer, Patrick E, and Gianluca Bontempi. 2006. “On the Use of Variable Complementarity for Feature Selection in Cancer Classification.” In Workshops on Applications of Evolutionary Computation, 91–102. Springer.
  
  
• Bennasar, Mohamed, Yulia Hicks, and Rossitza Setchi. 2015. “Feature Selection Using Joint Mutual Information Maximisation.” Expert Systems with Applications 42 (22): 8520–32.
  
  
• Bommert, Andrea, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang. 2020. “Benchmark for Filter Methods for Feature Selection in High-Dimensional Classification Data.” Computational Statistics & Data Analysis 143: 106839.
  
  
• Kursa, Miron B. 2021. “Praznik: High Performance Information-Based Feature Selection.” SoftwareX 16: 100819.