Feature Optimization Using Genetic Programming Lynca Kaminka, Pranjal Chalise, Stephen Chen, Theodore Woodward

Abstract In this paper, we propose the use of Genetic Programming (GP) algorithms to construct and refine feature sets for machine learning models. We develop two algorithms—one for feature extraction and one for feature selection—that systematically discover transformations of the original input data and subsets of features that enhance model performance. Our approach uses GP to navigate complex, high-dimensional search spaces without relying on domain-specific knowledge, thus offering a scalable and domain-agnostic solution to feature engineering. We evaluate our framework on multiple publicly available datasets, including the Heartbeat Categorization dataset from Kaggle and standard datasets from the scikit-learn repository (digits, wine, and breast cancer). Across a range of classification models (KNN, Random Forest, Logistic Regression, and SVM), we consistently observe improvements in classification accuracy. For instance, on the Heartbeat dataset, KNN’s baseline error was reduced from 0.2966 to 0.133 and SVM’s error decreased from 0.4157 to 0.215. On the Breast Cancer dataset, KNN’s error dropped from 0.062 to 0.021 and SVM’s from 0.094 to 0.031. Additionally, for the Digits dataset, GP-based feature extraction improved KNN’s baseline error from 0.012 to 0.008, Random Forest’s from 0.008 to 0.005, Logistic Regression’s from 0.01 to 0.006, and SVM’s from 0.006 to 0.005. These results demonstrate the significant potential of GP-based feature engineering in automatically uncovering complex data relationships that improve machine learning performance. By reducing reliance on manual, domain-intensive feature engineering efforts, our methods can streamline the model developme