Working with indels

[dc2211]

Hi all,

I’m trying to run the pipeline for a protein of length 400, and dms assay with varying length variants (between 394 to 406).

I’m using the —indel_mode argument, but I’m still receiving the error

RuntimeError: Received unaligned sequences for input MSA, all sequence lengths must be equal.

I created the MSA using the WT protein (400 residues).

Are there any other arguments to add for this to work?

Many thanks for the help

[pascalnotin]

Hi @dc2211 -- are you trying to run ProteinNPT on indels, or one of the other baselines (https://github.com/OATML-Markslab/ProteinNPT/tree/master/proteinnpt/baselines/model_configs)? The current codebase only supports indels with the "Embeddings" baselines (eg., https://github.com/OATML-Markslab/ProteinNPT/blob/master/proteinnpt/baselines/model_configs/Embeddings_ESM1v_final.json) but it is easy to adapt ProteinNPT to work with indels if you are interested in doing that.
Best,
Pascal

[dc2211]

Hi @pascalnotin ,

thanks for your answer! What Im trying to do is to run the pipeline, but training on my data. Specifically, my data looks like multiple sequences of different length (ranging from 394 to 406), derived from a WT of length 400. Now, I do have questions on how to properly set this up, because I would like to use the trained model to rank other candidates. Some of those candidate will have the same length of the WT (400), but maybe I will also include deletion or insertions in a similar range of the data that I use for training (394 to 406).

Now, when I built the MSA using EVcouplings, I used the WT sequence of 400 residues. This makes the MSA of length 400, and in here is where I believe the problem is, because my training data and sequence will not always have a length of 400. Not sure if this is actually possible right now, or Im just stuck on using ESM1v for inference. I do believe this is a zero-shot model, so not sure if I would be using the experimental data I have generated for this.

Please let me know if this makes sense. Many thanks

[pascalnotin]

Hi @dc2211 -- do you want to use MSA Transformer in the zero-shot setting to make predictions about sequences of varying lengths (ie indels), or do you want to use MSA Transformer to obtain protein embeddings, with which you will train supervised regressors on your experimental data?

For the former the starting point would be the zero-shot scripts we have in ProteinGym for MSA Transformer here: https://github.com/OATML-Markslab/ProteinGym/blob/main/scripts/scoring_DMS_zero_shot/scoring_MSA_transformer_substitutions.sh. It only handles substitutions at the moment (like for all other MLM baselines). It is possible though to adapt it to use the pseudo-perplexity heuristic to score indels.

For the latter, this should be currently supported by the code, but only for Embeddings MSAT (https://github.com/OATML-Markslab/ProteinNPT/blob/master/proteinnpt/baselines/model_configs/Embeddings_MSAT_final.json) not ProteinNPT (although it should not be too complex to adapt the code to do ProteinNPT as well).

[dc2211]

hi @pascalnotin ,

thanks again for the prompt response. What Im trying to do is more like the second option: using MSAT to obtain the embeddings that I will use to train a supervised reg model on my experimental data. The input I am using is the following:

export assay_data_location="/home/proteinnpt/data/amy_round1_indels.csv"
export MSA_location="/home/proteinnpt/data/msa_amy.a2m"
export target_seq="MGKNGSLCCFSLLLLLLLAGLASGHQVLFQGFNWESWKQSGGWYNMMMGKVDDIAAAGVTHVWLPPPSHSVSNEGYMPGRLYDIDASKYGNAAELKSLIGALHGKGVQAIADIVINHRCADYKDSRGIYCIFEGGTSDGRLDWGPHMICRDDTKYSDGTANLDTGADFAAAPDIDHLNDRVQRELKEWLLWLKSDLGFDAWRLDFARGYSPEMAKVYIDGTSPSLAVAEVWDNMATGGDGKPNYDQDAHRQNLVNWVDKVGGAASAGMVFDFTTKGILNAAVEGELWRLIDPQGKAPGVMGWWPAKAATFVDNHDTGSTQAMWPFPSDKVMQGYAYILTHPGIPCIFYDHFFNWGFKDQIAALVAIRKRNGITATSALKILMHEGDAYVAEIDGKVVVKI"
 
python pipeline.py \
    --proteinnpt_data_location ${proteinnpt_data_path} \
    --target_seq ${target_seq} \
    --assay_data_location ${assay_data_location} \
    --MSA_location ${MSA_location} \
    --fold_variable_name fold_random_5 \
    --test_fold_index 0 \
    --indel_mode \

amy_round1_indels.csv is a set of sequences of different length (394 to 406) derived from target_seq (length 400). The msa_amy.a2m was generated using the target_seq. By running this is where I encounter the issue that needs to have all the sequences in the same length as the MSA. Im not sure what else I am missing here, but indeed there is something that I still do not understand. Any support is greatly appreciated!

[pascalnotin]

Ok thanks for confirming! Could you please try running the embeddings script here first: https://github.com/OATML-Markslab/ProteinNPT/blob/master/scripts/embeddings_indels.sh ? This will help isolate the issue.

[dc2211]

do you mean literally running embeddings_indels.sh, or replacing with my data? if so, what are the arguments to update? I am seeing that this will mostly go through the already availble MSA and MSA weights in the ProteinNPT_data folders, but not sure if this aligns with what I have. To be clear, I had no issues generating the embeddings before, and the problem happened at the training step. Thanks!

[pascalnotin]

Once you have obtained the embeddings for all sequences you are interested in for training / inference, the MSA is not used anywhere else in the code, so I assumed that the issue was happening in the embedding creation stage.
One other thought: you should make sure to deactivate the computation of zero-shot scores in the pipeline script, as these are not currently supported with ESM1v / MSA Transformer for indels.

[dc2211]

I made sure that I was running the embedding generation on the correct file, as also making sure the MSA was properly generated using the WT sequence. I can confirm that all the sequences in the MSA have the same length as the WT, while the sequences we embed, coming from the experiments, have different lengths.

Unfortunately, Im still getting the same error, related that mutant sequences do not match with the length of the MSA. Attached is the output. I also double checked that batch['mutant_mutated_seq_pairs'] containst the mutant sequences, and some of them are shorter, which is expected since the experimental data I would like to use for training contains indel variants from the WT sequence.

I hope this helps to determine the source of the issue. Thanks!

Training ProteinNPT model on the amy_round1_indels assay
####################################################################################################
Step1: Computing sequence embeddings for the amy_round1_indels assay
####################################################################################################
Using embedding model: MSA_Transformer
Assay: amy_round1_indels.csv
4697+1 records in
4697+1 records out
2405170 bytes (2.4 MB) copied, 0.00878676 s, 274 MB/s
- 12:35:22.462 INFO: Input file = /home/ProteinNPT/AMY/preprocessed/msa_amy_UC.a2m


- 12:35:22.462 INFO: Output file = /home/ProteinNPT/AMY/hhfiltered/msa_amy_hhfiltered_cov_75_maxid_90_minid_0.a2m


Number of sequences in MSA (before preprocessing): 97
Calculating proportion of gaps
Proportion of sequences dropped due to fraction of gaps: 0.0%
Proportion of non-focus columns removed: 0.0%
Proportion of sequences dropped due to indeterminate AAs: 2.06%
Number of sequences after preprocessing: 95
Data Shape = (95, 400, 20)
Loading sequence weights from disk: /home/ProteinNPT/ProteinNPT_data/data/MSA/MSA_weights/hhfiltered/amy_round1_indels_theta_0.2.npy
Neff = 74.23333333333333
Number of sequences:  95
Embedding sequences: 100%|███████████████████████████████████████████████████████| 83/83 [1:14:01<00:00, 53.51s/it]
####################################################################################################
Step2: No zero-shot fitness predictions used for the amy_round1_indels assay
####################################################################################################
####################################################################################################
Step3: Training the ProteinNPT model on the amy_round1_indels assay
####################################################################################################
Location used for target fitness is: /home/ProteinNPT/AMY
Embeddings folder: /home/ProteinNPT/ProteinNPT_data/ESM/MSA_Transformer/esm_msa1b_t12_100M_UR50S.pt
We want to predict 1 target(s): fitness
Training model for assay: amy_round1_indels, where the test_fold index is: 0
Effective batch size is 425
Model name: ProteinNPT_amy_round1_indels_fitness_fold_random_5_embed_MSA_Transformer_head_CNN_aug_none_froz_True_drop_0.0_val_False_base_pipeline_fold-0
Sequence embeddings: /home/ProteinNPT/ProteinNPT_data/data/embeddings/indels/MSA_Transformer/amy_round1_indels.h5
Target processing train set: {'fitness': {'mean': 128.53920898815386, 'std': 87.69195208915613, 'P95': 251.08979881599997}}
Number of sequences in MSA (before preprocessing): 97
Calculating proportion of gaps
Proportion of sequences dropped due to fraction of gaps: 0.0%
Proportion of non-focus columns removed: 0.0%
Proportion of sequences dropped due to indeterminate AAs: 2.06%
Number of sequences after preprocessing: 95
Data Shape = (95, 400, 20)
Loading sequence weights from disk: /home/ProteinNPT/ProteinNPT_data/data/MSA/MSA_weights/hhfiltered/amy_round1_indels_theta_0.2.npy
Neff = 74.23333333333333
Number of sequences:  95
Starting training
Model device: cuda:0
  0%|                                                                                    | 0/10000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/ProteinNPT/scripts/train.py", line 460, in <module>
    main(args)
  File "/home/ProteinNPT/scripts/train.py", line 300, in main
    trainer_final_status = trainer.train()
                           ^^^^^^^^^^^^^^^
  File "/home/conda_envs/pnpt_env/lib/python3.12/site-packages/proteinnpt/utils/model_utils.py", line 166, in train
    processed_batch = proteinnpt.proteinnpt.data_processing.process_batch(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/conda_envs/pnpt_env/lib/python3.12/site-packages/proteinnpt/proteinnpt/data_processing.py", line 185, in process_batch
    batch_sequence_names, batch_AA_sequences, batch_token_sequences = token_batch_converter(batch['mutant_mutated_seq_pairs'])
                                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/conda_envs/pnpt_env/lib/python3.12/site-packages/proteinnpt/utils/esm/data.py", line 327, in __call__
    raise RuntimeError(
RuntimeError: Received unaligned sequences for input to MSA, all sequence lengths must be equal.
Traceback (most recent call last):
  File "/home/ProteinNPT/scripts/pipeline.py", line 317, in <module>
    subprocess.run("python train.py "+training_run_parameters, check=True, shell=True)
  File "/home/conda_envs/pnpt_env/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'python train.py 
--data_location /home/ProteinNPT/ProteinNPT_data         
--assay_data_location /home/ProteinNPT/AMY/amy_round1_indels.csv         
--model_config_location ../proteinnpt/proteinnpt/model_configs/PNPT_final.json         
--fold_variable_name fold_random_5         
--target_config_location ../proteinnpt/utils/target_configs/fitness.json         
--zero_shot_fitness_predictions_location /home/ProteinNPT/ProteinNPT_data/data/zero_shot_fitness_predictions/indels         
--training_fp16         
--sequence_embeddings_folder /home/ProteinNPT/ProteinNPT_data/data/embeddings/indels/MSA_Transformer         
--MSA_data_folder /home/ProteinNPT/AMY         
--MSA_location /home/ProteinNPT/AMY/msa_amy.a2m         
--MSA_weight_data_folder /home/ProteinNPT/ProteinNPT_data/data/MSA/MSA_weights         
--MSA_sequence_weights_filename amy_round1_indels_theta_0.2.npy         
--target_seq MGKNGSLCCFSLLLLLLLAGLASGHQVLFQGFNWESWKQSGGWYNMMMGKVDDIAAAGVTHVWLPPPSHSVSNEGYMPGRLYDIDASKYGNAAELKSLIGALHGKGVQAIADIVINHRCADYKDSRGIYCIFEGGTSDGRLDWGPHMICRDDTKYSDGTANLDTGADFAAAPDIDHLNDRVQRELKEWLLWLKSDLGFDAWRLDFARGYSPEMAKVYIDGTSPSLAVAEVWDNMATGGDGKPNYDQDAHRQNLVNWVDKVGGAASAGMVFDFTTKGILNAAVEGELWRLIDPQGKAPGVMGWWPAKAATFVDNHDTGSTQAMWPFPSDKVMQGYAYILTHPGIPCIFYDHFFNWGFKDQIAALVAIRKRNGITATSALKILMHEGDAYVAEIDGKVVVKI         
--test_fold_index 0         
--augmentation None         
--model_name_suffix base_pipeline 
--MSA_start 1 
--MSA_end 400 
--indel_mode' 

returned non-zero exit status 1.

[dc2211]

Hi all, following up on this thread. Your help is greatly appreciated. Thanks!

[pascalnotin]

Hi @dc2211 -- I pushed a new version (1.5.5) which I have used for a different project where we had to deal with sequences of varying length (as discussed here: https://github.com/OATML-Markslab/ProteinNPT). Let me know if that addresses the issues you were facing. Otherwise happy to have a closer look at it if you share your data with me (feel free to anonymize / scramble numeric values as needed!).
Best,
Pascal