Accuracy using pretrained weights is 0.69

[zhanxw]

The frozen_embeddings_classification.ipynb reported that the accuracy is 0.86.
I rerun this notebook on two servers, and both have accuracy around 0.69. (notebook attached).

Has anyone else noticed a similar decrease in accuracy?

replicate.ipynb.zip

[mtinti]

if I use:
test_df = pd.read_csv('../../data/bace/test.csv')
insted of:
test_df = pd.read_csv('../../data/bace/test.csv').sample(frac=0.1)

I get ROC 0.8374908290535583

[xdn-github]

Yey, I got 0.69 too,

What's more, this pre-train model was trained on the 10%Zinc+10%PubChem as described in README(fig shonw below), which is NOT the best model trained on 1.1B datasets.

BUT, even for this pre-trained model, it shoud have 86.6 on BACE test. Scores from the preprint version assay as shown below

I don't know which step I got wrong..