Not all databanks are indexed (correctly)

[drlemmus]

A number of databanks do not have any statistics (e.g. HSSP and PDB_REDO). The counts for other databanks (e.g. DSSP and STRUCTURFACTORS) are incorrect: missing entries are not listed.

[jonblack]

Coos is looking into this for #38.

[jonblack]

#38 is now closed. The stats page is now correct based on what's in the database. The issue is with the crawler and annotator.

[jonblack]

The crawling/annotation method is a bit backwards. We start the process without any expectations. If we scan only 100 PDB files, then we only expect 100 files maximum in the other databanks that depend on the PDB.

In reality we have a pretty good idea before we start what the ideal scenario is. We can download a list of all valid and obsolete PDB IDS from pdb.org and use that as a base. When we crawl we're no longer indexing what we have but instead checking to see what's missing. Those that are missing can then be passed through the annotator.

I'm going to update this process to use the ids downloaded from pdb.org as the source.