Mapping protein IDs from NP to ENSP

[bschilder]

Hi @pascalnotin

This related to my previous issue (#61 ), but I have a more specific request.
Could you possibly provide a 1:1 mapping between all of your RefSeq protein IDs (NP_xxxxx.y) and Ensembl protein IDs (ENSPxxxxx.y)?

I've tried doing the mapping myself but I'm afraid there's simply too many possible many:many mappings that make this process ambiguous. The end result is that the sequence I'm getting from Ensembl API is not matching up with the sequence in the preprocessed clinical sub/indel mutation provided by ProteinGym. It's a little hard for me to pinpoint exactly where the issue is stemming from without the ID mappings.

Thanks in advance,
Brian

[loodvn]

Hi Brian, thanks for the questions and apologies for the delay.

Completely understand the mapping woes - Aaron and I took another look at our original notebook and uploaded it here (https://marks.hms.harvard.edu/proteingym/other/clinical_subs_stats_internal_20250226.ipynb).

The raw output file with all original columns from both dbNSFP and the Ensembl VEP annotation is at https://marks.hms.harvard.edu/proteingym/other/all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv. The column "protein" is the RefSeq protein ID (NP_) and then we kept Ensembl_proteinid and the other Ensembl IDs there.

Let me know if that clears up some of the mapping issues or if you have further questions. Particularly the deduplication of multiple isoforms was a bit tricky as you might see.

[bschilder]

Hey @loodvn thanks for these, this is a huge help! I'll have a go and let you know what I find.

I can provide a report on which IDs I get AA sequence mismatches relative to the data I'm pulling from Ensembl, if that's helpful.

[loodvn]

That would be useful thanks!

[bschilder]

Import the raw data and create a mapping df:

pg_map = pg.map_proteingym_ids(return_map=True)
pg_annot = pd.read_csv('https://marks.hms.harvard.edu/proteingym/other/all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv')
select_cols = ['protein',
                'Ensembl_geneid','Ensembl_transcriptid','Ensembl_proteinid']
pg_annot = pg_annot[select_cols].drop_duplicates()
pg_annot

Count the number of ENS* IDs per RefSeq protein ID:

summary_df = pd.DataFrame()
for col in ['Ensembl_geneid', 'Ensembl_transcriptid', 'Ensembl_proteinid']:
    # Get nunique counts per protein
    nunique_counts = pg_annot.groupby('protein')[col].nunique()
    
    # Get basic stats
    stats = nunique_counts.describe()
    
    # Add count of proteins with >1 ID
    stats.loc['proteins_with_multiple'] = (nunique_counts > 1).sum()
    
    # Add percentage of proteins with >1 ID
    stats.loc['proteins_with_multiple_percent'] = (nunique_counts > 1).mean() * 100
    
    # Add to summary df
    summary_df = pd.concat([summary_df,
                          pd.DataFrame({'col': col,
                                      'values': stats})])
summary_df.reset_index(inplace=True)
summary_df

index	col	values
count	Ensembl_geneid	2514.000000
mean	Ensembl_geneid	1.005569
std	Ensembl_geneid	0.079598
min	Ensembl_geneid	0.000000
25%	Ensembl_geneid	1.000000
50%	Ensembl_geneid	1.000000
75%	Ensembl_geneid	1.000000
max	Ensembl_geneid	2.000000
proteins_with_multiple	Ensembl_geneid	15.000000
proteins_with_multiple_percent	Ensembl_geneid	0.596659
count	Ensembl_transcriptid	2514.000000
mean	Ensembl_transcriptid	1.211217
std	Ensembl_transcriptid	0.476609
min	Ensembl_transcriptid	0.000000
25%	Ensembl_transcriptid	1.000000
50%	Ensembl_transcriptid	1.000000
75%	Ensembl_transcriptid	1.000000
max	Ensembl_transcriptid	5.000000
proteins_with_multiple	Ensembl_transcriptid	462.000000
proteins_with_multiple_percent	Ensembl_transcriptid	18.377088
count	Ensembl_proteinid	2514.000000
mean	Ensembl_proteinid	1.211217
std	Ensembl_proteinid	0.476609
min	Ensembl_proteinid	0.000000
25%	Ensembl_proteinid	1.000000
50%	Ensembl_proteinid	1.000000
75%	Ensembl_proteinid	1.000000
max	Ensembl_proteinid	5.000000
proteins_with_multiple	Ensembl_proteinid	462.000000
proteins_with_multiple_percent	Ensembl_proteinid	18.377088

So about 18% of the RefSeq protein IDs in the curated ProteinGym data have multiple ID mappings to Ensembl Transcripts and Proteins.

This is a helpful baseline expectation for me when comparing the actual sequences. Will report that next.

[Joey-Xue]

Hello Brian @bschilder and Lood @loodvn Thanks for your kind effort for raising the issue and providing the code, I'm also interested in mapping the protein sequences to ensembl ids or further genome coordinates, but I got 2 issues accessing the mapping:

1. Neither of the file at https://marks.hms.harvard.edu/proteingym/other/clinical_subs_stats_internal_20250226.ipynb nor https://marks.hms.harvard.edu/proteingym/other/all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv is accessible through the link, would you mind sharing it again?
2. I think 'pg' in your code refers to proteingym package on pywheels https://www.piwheels.org/project/proteingym/
   But after I installed it and try import proteingym as pg I got error
   maybe there's something that I misunderstood?

`File ~/anaconda3/envs/protein_py39/lib/python3.9/site-packages/proteingym/init.py:1
----> 1 from proteingym import utils

ImportError: cannot import name 'utils' from partially initialized module 'proteingym' (most likely due to a circular import) (/Users/zyxue/anaconda3/envs/protein_py39/lib/python3.9/site-packages/proteingym/init.py)`

Thanks again for your kind sharing and look forward to hearing back from you!
Best,
Joey

[ElainW]

Hi @Joey-Xue, the .ipynb file and the .csv files should be available now through the links you posted. Let me know if you have issues accessing them :)

[bschilder]

2. I think 'pg' in your code refers to proteingym package on pywheels https://www.piwheels.org/project/proteingym/
   But after I installed it and try import proteingym as pg I got error
   maybe there's something that I misunderstood?
   `File ~/anaconda3/envs/protein_py39/lib/python3.9/site-packages/proteingym/init.py:1 ----> 1 from proteingym import utils

ImportError: cannot import name 'utils' from partially initialized module 'proteingym' (most likely due to a circular import) (/Users/zyxue/anaconda3/envs/protein_py39/lib/python3.9/site-packages/proteingym/init.py)`

Thanks again for your kind sharing and look forward to hearing back from you! Best, Joey

@Joey-Xue pg is actually a custom module I wrote to download and manipulate ProteinGym data. I'm afraid I can't share it just yet as it's an unpublished work and the code base is too complex to extract just that portion and have it still work. But essentially pg_map = pg.map_proteingym_ids is just importing the processed clinical sub/indel tables from ProteinGym:

https://marks.hms.harvard.edu/proteingym/ProteinGym_v1.3/clinical_ProteinGym_substitutions.zip
https://marks.hms.harvard.edu/proteingym/ProteinGym_v1.3/clinical_ProteinGym_indels.zip

...and then merging them with the all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv shared above file.

[Joey-Xue]

The files are available now, thanks a lot Yilan and Brian!

[bschilder]

Hi Brian, thanks for the questions and apologies for the delay.

Completely understand the mapping woes - Aaron and I took another look at our original notebook and uploaded it here (https://marks.hms.harvard.edu/proteingym/other/clinical_subs_stats_internal_20250226.ipynb).

The raw output file with all original columns from both dbNSFP and the Ensembl VEP annotation is at https://marks.hms.harvard.edu/proteingym/other/all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv. The column "protein" is the RefSeq protein ID (NP_) and then we kept Ensembl_proteinid and the other Ensembl IDs there.

Let me know if that clears up some of the mapping issues or if you have further questions. Particularly the deduplication of multiple isoforms was a bit tricky as you might see.

Hey @loodvn , did you guys happen to change the mapping file (https://marks.hms.harvard.edu/proteingym/other/all_models_deduplicated_scores_clinvar_proteingym_20230611.GRCh38_filter_isoform_clean.csv) since v1.1? Unlike the other files it doesnt seem that this one is version-controlled. As a result, I seem to be missing some columns my pipeline relies on:

['genename', 'Ensembl_geneid', 'Ensembl_transcriptid', 'Ensembl_proteinid']

There also used to be UniProt IDs, though those are less essential for me atm:
'Uniprot_acc','Uniprot_entry','Uniprot_acc(HGNC/Uniprot)','Uniprot_id(HGNC/Uniprot