how to handle the category columns

[chenlei76]

For example, I have the following data:
X = pd.DataFrame({
"numerical_col": [1.2, 3.5, np.nan, 7.8, 9.1],
"categorical_col": ["A", "B", "B", "C", "A"],
"integer_col": [10, 20, 30, 40, 50]
})
I'm working on a numerical prediction task, but my input data includes categorical columns. Is there a quick, ready-to-use interface that I can directly utilize?

Further, if my prediction targets are categorical variables, is there a quick, out-of-the-box interface that I can use directly?
For example, I have the following data:
X = pd.DataFrame({
"numerical_col": [1.2, 3.5, np.nan, 7.8, 9.1],
"categorical_col": ["A", "B", np.nan", "C", "A"],
"integer_col": [10, 20, 30, 40, 50]
})

Thanks!

[chenlei76]

I just found TabImputeCategorical in the project, and directly used TabImputeCategorical for
X = pd.DataFrame({
"numerical_col": [1.2, 3.5, np.nan, 7.8, 9.1],
"categorical_col": ["A", "B", "B", "C", "A"],
"integer_col": [10, 20, 30, 40, 50]
})
There exits a error: ValueError: could not convert string to float: 'A'
Does it mean that I need to manually use one-hot encoder to convert each category column? Further, if I want to impute the missing category values, how? Thanks.

[jacobf18]

Hi @chenlei76 I just got your examples to work with the categorical imputation class by converting the X.to_numpy() in the impute() function. The function works on numpy arrays only right now, so that is a quick work around. I will add better support to pandas dataframes soon, but this numpy conversion should work properly.

[jacobf18]

The class uses one-hot encoding, runs the fully numerical matrix through the model, and then converts these imputed encodings back to categories using softmax and choosing the class with the highest predicted probability.

[chenlei76]

Thanks for your quick reply!
I noticed that TabPFN claims to support datasets with up to 10 million rows. However, when I tried creating a dataset of shape (50, 200) and calling the impute() method of TabImputeCategorical, my 125GB of RAM was quickly exhausted, and the process eventually got killed.

I then tested with a smaller dataset of shape (30, 200). The impute() method ran successfully, but it still took as long as 1300 seconds to complete.

My actual requirement is to impute missing values for a dataset with 500 rows and 200 columns. I was wondering if you have conducted experiments with this scale of data. Could you please share approximately how much GPU VRAM is required and what the expected runtime would be?

[jacobf18]

@chenlei76 Yes, we will soon realease version 2 of our model that uses a different architecture and is way faster for bigger matrices.

[jacobf18]

@chenlei76 If you upgrade the version of the package on pip, you can use the new model via:

from tabimpute.tabimpute_v2 import TabImputeV2

imputer = TabImputeV2(device='cpu')
imputer.impute(X.copy())

This model should run a lot faster and has much lower memory usage. We haven't yet updated the categorical class to use the new model, but will do that soon. However, note that the categorical class uses one-hot encoding for categorical columns, so if those columns have a lot of classes, then it adds 1 column per feature, which could be a lot. We are planning on working on models which handle categorical variables natively in the future.