Skip to content

[NEW SUBMISSION] Confidence-Calibrated Delta scoring function#100

Open
EmmettBicker wants to merge 1 commit intoOATML-Markslab:mainfrom
EmmettBicker:prosst-2048-single-pass-delta
Open

[NEW SUBMISSION] Confidence-Calibrated Delta scoring function#100
EmmettBicker wants to merge 1 commit intoOATML-Markslab:mainfrom
EmmettBicker:prosst-2048-single-pass-delta

Conversation

@EmmettBicker
Copy link
Copy Markdown

@EmmettBicker EmmettBicker commented Mar 10, 2026

This PR adds in a new scoring method called Confidence-Calibrated Delta for ProSTT (K=2048) which raises the avg spearman on DMS_zero_shot from 0.507 to 0.523.

The intuitive change is going from

score = mutant score − wildtype score

to

score = mutant score − background penalty − uncertainty penalty + wildtype confidence bonus

Or more mathematically,

Screenshot 2026-03-09 at 11 13 22 PM

to

Screenshot 2026-03-09 at 11 16 43 PM

where l(a) = log p(a) and where background_weight and wildtype_confidence_weight are hyperparameters set to 0.35 and 0.2 respectively.

Here were my results:

  • Average Spearman: 0.523
  • Average NDCG: 0.773
  • Average AUC: 0.785
  • Average MCC: 0.408
  • Average Top recall: 0.237

Selected Spearman breakdowns:

  • Activity: 0.507
  • Binding: 0.455
  • Expression: 0.542
  • Organismal fitness: 0.451
  • Stability: 0.659
  • Low MSA depth: 0.512
  • Medium MSA depth: 0.517
  • High MSA depth: 0.590

Takes ~3 minutes to run

I also independently validated average spearkman of K=1024 and K=4096

  • 1024: 0.491
  • 4096: 0.503
    But, I'm only interested in submitting 2048.

@EmmettBicker EmmettBicker changed the title Add ProSST-2048 single-pass delta baseline [NEW SUBMISSION] Altered scoring function results in significant boost for ProSST models. Applied to ProSST 2048, this tops the benchmark Mar 10, 2026
@EmmettBicker EmmettBicker changed the title [NEW SUBMISSION] Altered scoring function results in significant boost for ProSST models. Applied to ProSST 2048, this tops the benchmark [NEW SUBMISSION] Confidence-Calibrated Delta scoring function Mar 10, 2026
@EmmettBicker
Copy link
Copy Markdown
Author

Hey @pascalnotin -- When you have time, would really appreciate a review of this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant