No tmp outfile error

[mdhar1980]

Hello, Benjamin,

Hi, I tweeted at you about our issue with running the conversion of a FASTA file command. We used a command exactly modeled on your sample, but get this error. We have also tried modifcations and did the normal things like double-check filepaths:

ERROR:
Traceback (most recent call last): File "/home/biotools/brisera-master/apps/convert_fasta.py", line 40, in for name in os.listdir(tempout):OSError: [Errno 2] No such file or directory: '/tmp/fastayIV9aq/output'

This is our command:

spark-submit --master local[*] /home/biotools/brisera-master/apps/convert_fasta.py /home/ajm/Staphdata/RefSeq/genome/GCF_000284535.1_ASM28453v1_genomic.fna /home/ajm/outputs/brisera/

And with the file paths simplified to working directory within Brisera:
spark-submit --master local[*] apps/convert_fasta.py input.fa output.ser

Thank you!
---Michael Dhar
_

[bbengfort]

Well, that is interesting. On L30 I'm creating a temporary directory using Python's tempfile. I'm then expecting PySpark to create the output directory inside of that (rather than creating it myself). I guess Spark isn't doing that for some reason?

My suggestion is to add the following to convert_fasta.py at line 32:

if not os.path.exists(tempout):
    os.makedirs(tempout)

This will ensure the directory is created. Not sure why Spark is no longer doing it on your behalf.

[mdhar1980]

Thanks much! We'll give that a try