Issues with current ML validation score

[GinoWoz1]

Hello,

Thanks for the help so far. I was able to get the tool up and running in windows.

However, 2 weird things I am observing.

1. When I use Gradient Boost Regressor - my score gets worse by the generation even when I switched the scoring function sign. The first score is nearly my best score I have gotten by myself (no feature engineering done on data set).

https://github.com/GinoWoz1/AdvancedHousePrices/blob/master/FEW_GB.ipynb

2. When I use Random Forest - same scorer - current ML validation score returns as 0 and runs really fast

https://github.com/GinoWoz1/AdvancedHousePrices/blob/master/FEW_RF.ipynb

I think I am missing something on how to use this tool but no idea what. I am trying to use this in tandem with TPOT as I am exploring feature creation GA/GP based tools. Sincerely appreciate any advice/guidance you can provide.

Sincerely,
G

[GinoWoz1]

Hello @lacava

Sorry for the bother , but have you had a chance to look at this ? I have been messing around with TPOT for the last 4 months and have talked to Randy Olson a few times ; he had referred me to Few and I am hoping to do a few tests with Few and Tpot over the winter . My name is Justin Joyce and currently I am exploring multiple genetic algorithm and programming methods as a masters student .

Sincerely,
Justin

[lacava]

Hi Justin, I did look at it and ran it a couple times. It looks like there is a small bug with Few, which is that it prints out that the current ml validation score is 0 when it is not, as shown by the internal CV score that is printed.

Otherwise, this just seems to be a dataset that is not amenable to feature learning. I have found that, when paired with Gradient boosting or other high-capacity methods, it is quite difficult to find a transformation of the data that will improve the underlying ML using Few. Using Lasso, I was able to occasionally find a reduced feature space, but not one that dramatically improved the score.

You also may be interested in trying Feat, which is a more powerful version of Few that I have been working on for the last year. It has a similar sklearn interface, uses a GA to drive search, and includes neural network activation functions and backprop for learning weights. Here's the result of running that:

from feat import Feat
from sklearn.metrics import r2_score


learner = Feat(gens=1000,max_stall=100,pop_size=100,backprop=True,
               verbosity=2,
               max_dim=50,
               feature_names = ','.join(X_train.columns))
X = X_train.values
y = y_train
learner.fit(X,y)


print('final score: {}'.format(r2_score(y_train, learner.predict(X)))) 
  
print('model:\n',learner.get_model())

final score: 0.9004700679745707
model:
Feature Weight
relu(2ndFlrSF) 4575965.203086
(2ndFlrSF^2) -2993884.266219
(2ndFlrSF^3) 2704420.880597
2ndFlrSF -2653911.531777
relu(2ndFlrSF) -2328605.023701
(GrLivArea^3) -972888.992427
LotArea 829523.137047
YearBuilt 741179.594699
relu(GrLivArea) 689061.927277
relu(OverallQual) 630256.458467
relu(LotArea) -593285.483499
TotalBsmtSF 536721.385827
float(OverallCond) 408904.496879
sqrt(|YearBuilt|) 341173.116787
1stFlrSF 340342.422143
float(GarageCars) 286630.673027
OverallQual 230831.115685
(OverallQual*GarageArea) 214190.124423
BsmtFinSF1 192567.902371
(TotalBsmtSF+BsmtUnfSF) -192281.015785
relu(OverallQual) 189096.541817
float(Fireplaces) 183822.299462
GrLivArea 155897.078241
float(Condition2_Norm) 116659.048371
ScreenPorch 102781.956623
float(Neighborhood_OldTown) -100922.111480
float(HalfBath) 97458.382917

The downside is that you can't specify your own scoring_function at the moment.

[lacava]

When I use Gradient Boost Regressor - my score gets worse by the generation even when I switched the scoring function sign. The first score is nearly my best score I have gotten by myself (no feature engineering done on data set).

This i did not observe. I did observe that Few did not find better features, but the Internal CV stayed constant, as it should.