`NewCRM_Prediction.py` fails to predict the simplest textbook Diels-Alder reaction

[yaroslavsobolev]

In your preprint, you mention that

In addition, the inclusion of Diels–Alder cycloaddition reactions brings a desirable mechanistic diversity in the form of pericyclic reactions.

So your training set is described as containing Diels-Alder cycloadditions. However, NewCRM_Prediction.py fails to predict the classical Diels-Alder reaction between 1,3-butadiene and ethene: the results is an empty list. The expected outcome is cyclohexene. Here is the end of the NewCRM_Prediction.py that I ran:

start_sequence = "C=CC=C.C=C"
results = beam_search_last_element2(start_sequence, model_predict, beam_width=2, max_steps=10, alpha=0.5)
print('RESULTS:')
print(results)

The output is:

RESULTS:
[]

It does not seem to be a problem with my installation because your much more complex example of start_sequence in your repository works correctly for me, as expected:

start_sequence = "Cc1c(Cl)cccc1.CN(c2ccc([P+]([C@]34C[C@H]5C[C@@H](C4)C[C@@H](C3)C5)([C@]67C[C@H]8C[C@@H](C7)C[C@@H](C6)C8)[Pd])cc2)C.CC(C)([O-])C.CCCCN.[Na+]"
results = beam_search_last_element2(start_sequence, model_predict, beam_width=2, max_steps=10, alpha=0.5)
print('RESULTS:')
print(results)

Is the failure to predict the simplest Diels-Alder some simple bug of formatting that can be fixed? Or is it a deeper problem with the entire pipeline?

P.S.: In the current state of the repository, the unused method calculate_top_k_accuracies should be removed from the import statement in line 19 of file NewCRM_Prediction.py, otherwise it causes an import error.

[manajit-das]

Thank you very much for taking the time to report this issue and for providing a clear minimal example.

DeepMech was trained on a corpus of 104,964 elementary steps, among which only 100 correspond to Diels–Alder cycloaddition reactions. This makes the dataset highly imbalanced, with the Diels–Alder class contributing only a very small portion of the training distribution. Moreover, the available Diels–Alder examples in our dataset predominantly involve heavily substituted dienes and dienophiles (examples are listed in TwoBondChem/dataset/da_08012024_vCorr.csv at main · Goodman-lab/TwoBondChem). As a result, simple textbook reactions, such as the cycloaddition between 1,3-butadiene and ethane, fall outside the narrow distribution of the Diels–Alder subset.

This effect becomes apparent when we test reactions involving only mild substitution. For example, DeepMech correctly predicts the outcomes for:

1.  1,3-butadiene + methyl vinyl ketone
   

2.  1,3-butadiene + methyl acrylate
   

3.  1-methoxy-1,3-butadiene + methyl acrylate
   

These reactions are structurally simple but still closer to the substituted examples present in the training set, even though these specific three examples were not themselves included in the training data. If you have not already done so, please increase the collect_n parameter in the predict_product function (inside the model_predict function) to around 200.

To better understand the failure mode for the butadiene–ethene case, we examined the raw model outputs (predicted bond changes and templates). Interestingly, it contains valid templates, for example:
"[A:1]=[A:2].[A:3]=[A:4]-[A:5]=[A:6]>>[A:1]1-[A:2]-[A:3]-[A:4]=[A:5]-[A:6]-1"
However, possibly lack of prediction of all the bonds as required to satisfy the template appear to prevent the correct application of these templates, leading to an empty result list in this instance. As with any deep-learning-based model, the quality and generality of the predictions are directly tied to the diversity and coverage of the training data.

We appreciate your interest in using DeepMech for your research and are actively looking into improvements for low-coverage reaction classes. Your feedback is extremely valuable for guiding those efforts.

Finally, thank you for pointing out the unused import of calculate_top_k_accuracies in NewCRM_Prediction.py. We will remove this in the next update.

Simple textbook reactions can be incorporated easily starting with atom mapped elementary steps. Our supporting information provides a step-by-step procedure on how to prepare and train DeepMech with your own data. The manuscript describing this process is currently under review and will be made available upon publication. We will also update the repository accordingly.

[manajit-das]

Upon closer inspection, we found that the failure to predict the product for the reaction between 1,3-butadiene and ethene originated in the post-processing step. We have now corrected the code, and the model successfully predicts the product structure. However, we also observed that during beam search the reaction classifier incorrectly labels the predicted product as reactive (i.e., requiring further steps), causing DeepMech to continue predicting beyond the intended endpoint. For this reason, we recommend that users also perform step-by-step predictions with DeepMech, in parallel with the beam-search workflow that relies on the reaction classifier.