Seems really difficult to reproduce the results

[mahi83]

I have been trying to replicate the default code you all set, where we want to predict readmission on "5-folds". I have run this scrip a couple of times on 1 million steps on "Fold 0". For some reason, I can't get any AUC-PR score near the reported score. I have been getting an AUC-PR of around 0.3. However, the paper reports an AUC score of around 0.5244. Any comments of changes I need to make to get similar results?

[jonasbkemp]

@mp2893

[mp2893]

Hi mahi83,

Thanks for trying out the code.
Let me ask a couple of questions first:

• For you experiments, did you use the exact same hyperparameters as in the paper Appendix?
• Did you get the same AUC-PR for all other folds?

Best,
Ed

[mahi83]

I used the same hyper parameters as mentioned in the appendix of the paper.

I was running into errors when running other folds. I didn't spend time trying to debug the errors, as I was focused on the first fold.

[foadgr]

Hi @mp2893 I'm also receiving errors running on Fold >= 1 using the paper's hyperparameters for the readmission prediction task. In Fold 0, my AUC-PR (~0.3) and AUC-ROC (~0.6) are both seem lower than in the published results. Running Fold 0 again now and here are metrics around ~160k steps.

[mp2893]

Hi @foadgr, I completely forgot about this issue.
To the best of my recollection, when I was looking into this issue a couple years ago, I think I found the source of this problem to be the preprocessing code for eICU, which was a new cleaner code I'd written for open-sourcing this work on GitHub.
Unfortunately, I don't remember the exact nature of the problem. At the time I had a suspicion that something was wrong with building the SequencExamples data structure, but didn't have the time to dig deep to verify the suspicion.
Unless you are super determined to use GCT for readmission prediction, I'd like to suggest that you try out my new approach, which is way simpler than GCT, and doesn't require as much preprocessing (hence fewer working parts that could go wrong).

[foadgr]

Sounds great, will give this a try. Thank you!