Integration with Canadian data portal

[ArtPoon]

Refactored local.py into two separate scripts.
One script is used to combine and filter data into an xz-compressed JSON file, or append to an existing file.
A second script is used with this xz-JSON as the primary input for analysis.

python3 convert.py working/n1e4.fasta.xz --xz working/n1e4.pangolin.csv working/metadata.tsv --delimiter '\t' --name strain --outfile test.json.xz

may equivalently be run as:

python3 convert.py working/n1e4.fasta.xz --xz working/n1e4.pangolin.csv working/metadata.tsv --delimiter '\t' --name strain | xz > test.json.xz

[ArtPoon]

Encountering some significant problems with the metadata TSV from the data portal:

• no region (North America) or country (Canada) fields
• there is no label that can be used to match rows in metadata table to sequence records in the FASTA file

[ArtPoon]

Country and sequence labels are available as optional fields (see Columns drop-down).

[ArtPoon]

Can now append to JSON xz file:

art@Wernstrom covizu % python3 convert.py working/n1e4.fasta.xz --xz working/n1e4.pangolin.csv working/metadata.tsv --delimiter '\t' --name strain --outfile test.json.xz
🏄 [0:00:30.729376] Rejected 16 short genomes, 1263 records with bad dates
art@Wernstrom covizu % unxz -c test.json.xz | wc -l
    8721
art@Wernstrom covizu % gunzip -c working/virusseq-consensus-export-2021-05-25T17_26_15.259802Z.fasta.gz | grep -c ">"
20
art@Wernstrom covizu % python3 convert.py working/virusseq-consensus-export-2021-05-25T17_26_15.259802Z.fasta.gz working/virusseq-pangolin-2021-05-25.csv working/virusseq-metadata-export-20210526.tsv --gzip --delimiter '\t' --name Isolate --coldate 'Sample Collection Date' --region Region --country Country --append test.json.xz
🏄 [0:00:00.082242] Rejected 0 short genomes, 0 records with bad dates
art@Wernstrom covizu % unxz -c test.json.xz | wc -l
    8741

[ArtPoon]

Oops - serialized Python dicts are not equivalent to JSON after all because of the single quotes:

art@orolo:~/git/covizu$ python3 process.py test.json.xz --outdir working/iss282/ 
Traceback (most recent call last):
  File "process.py", line 153, in <module>
    analyze_feed(handle, args, callback=cb.callback)
...
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Fixed.

[ArtPoon]

🏄 [0:01:41.084760] start B, 327 entries
🏄 [0:01:57.955842] start A, 257 entries
🏄 [0:02:05.278295] start B.1, 2924 entries
--------------------------------------------------------------------------
Open RTE was unable to open the hostfile:
    mfile
Check to make sure the path and filename are correct.
--------------------------------------------------------------------------
Traceback (most recent call last):
  File "process.py", line 158, in <module>
    analyze_feed(handle, args, callback=cb.callback)
  File "process.py", line 125, in analyze_feed
    result = make_beadplots(by_lineage, args, callback, t0=timestamp)
  File "/home/art/git/covizu/covizu/utils/batch_utils.py", line 103, in make_beadplots
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mpirun', '--machinefile', 'mfile', 'python3', 'covizu/clustering.py', 'data/by_lineage.json', 'B.1', '--nboot', '100', '--outdir', 'data', '--timestamp', '2021-05-26T20:57:00']' returned non-zero exit status 1.

Should have an option for running without an MPI environment.

[ArtPoon]

Disabled MPI by default.
Ran into another problem:

art@orolo:~/git/covizu$ python3 process.py test.json.xz --outdir working/iss282/ --ft2bin fasttree2-mp
🏄 [0:00:00.845342] aligned 0 records
...
🏄 [0:09:04.698638] start B.1.468, 2 entries
Traceback (most recent call last):
  File "process.py", line 163, in <module>
    analyze_feed(handle, args, callback=cb.callback)
  File "process.py", line 130, in analyze_feed
    result = make_beadplots(by_lineage, args, callback, t0=timestamp)
  File "/home/art/git/covizu/covizu/utils/batch_utils.py", line 92, in make_beadplots
    beaddict = beadplot_serial(lineage, features, args)
  File "/home/art/git/covizu/covizu/utils/batch_utils.py", line 38, in beadplot_serial
    for coldate, accn, label1 in intermed:
ValueError: not enough values to unpack (expected 3, got 2)

[ArtPoon]

Fixed a problem with running MPI with -np flag.
Much faster even with 6 cores.
Reproduced error with some more info using this modification to batch_utils.py:

        try:
            for coldate, accn, label1 in intermed:
                beaddict['nodes'][variant].append([coldate, accn, label1])
        except ValueError:
            print(intermed)
            raise

🏄 [0:03:42.880638] start B.1.468, 2 entries
[['2020-02-13', 'SAU/545/2020'], ['2020-02-18', 'SAU/596/2020']]

[ArtPoon]

Successful run of backend with commit ca7a0a5
Front end fails to parse data files:

Uncaught TypeError: Cannot read property 'replace' of undefined
    at parse_variant (beadplot.js:153)
    at parse_clusters (beadplot.js:380)
    at Object.<anonymous> (covizu.js:150)
    at fire (jquery.js:3496)
    at Object.fireWith [as resolveWith] (jquery.js:3626)
    at done (jquery.js:9786)
    at XMLHttpRequest.<anonymous> (jquery.js:10047)

[ArtPoon]

Need to load country annotations into JSON data

[ArtPoon]

For some reason district keys are not being carried over from by_lineage.json:

🏄 [0:01:39.482498] start B, 327 entries
Traceback (most recent call last):
  File "process.py", line 163, in <module>
    analyze_feed(handle, args, callback=cb.callback)
  File "process.py", line 130, in analyze_feed
    result = make_beadplots(by_lineage, args, callback, t0=t0.timestamp())
  File "/home/art/git/covizu/covizu/utils/batch_utils.py", line 93, in make_beadplots
    beaddict = beadplot_serial(lineage, features, args)
  File "/home/art/git/covizu/covizu/utils/batch_utils.py", line 28, in beadplot_serial
    trees, labels = clustering.build_trees(features, args, callback=callback)
  File "/home/art/git/covizu/covizu/clustering.py", line 217, in build_trees
    union, labels, indexed = recode_features(records, callback=callback)
  File "/home/art/git/covizu/covizu/clustering.py", line 34, in recode_features
    label = '{label}|{region}|{country}|{district}|{coldate}'.format(**record)
KeyError: 'district'

[ArtPoon]

Gah! keyword is division, not district

[ArtPoon]

Running into some difficults with front-end because we stopped requiring sample accessions in the data.
We were using accession numbers as unique identifiers for SVG elements.

Also running into this problem:

drawtree.js:277 Uncaught TypeError: mutations is not iterable
    at mutations_to_string (drawtree.js:277)
    at mouseover (drawtree.js:324)
    at SVGTextElement.<anonymous> (drawtree.js:416)
    at SVGTextElement.<anonymous> (d3.js:1526)

For some reason tips are not getting annotated with mutations.

[ArtPoon]

We'll continue to try to modify the front-end to remove all dependency on sample accession numbers, and assume that sequence names are unique (need to add a check for this in backend).
That means that sequence names are re-used as element IDs, in addition to being text labels in some cases.

[ArtPoon]

If we omit accession numbers then these are not available to the user via the front-end.