Initial pull request for the first prototyped rapid data generating pipeline #23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…and one for performing the downloads. The snakefile has been adapted to incorporate both of these steps into the procedure
…t expects directory output.
…eparate jobs to download cds, pep and gtf files. Each job uses the new downloader.sh script, using diffferent input arguments
…eny matrix. Scripts used to use dir_name of data_homology for homology databases. I have renamed this to homology_databases and modified the python scripts to work with the new naming convention
…way up to the run process_negative.py scripts successfully
…o. Date: Sat Jun 12 07:27:00 BST 2021
…ifferent databases and supply those download links
…at not all databases have the same sets of species available. New snakemake re-config and and adaptation of the ftpy.py script. The user should now run ftpy.py before running the snake file. After that point, the pipeline should be determinitistic
…s should be a list of the species which are in the intersection of all the different input databases
…jobs kept being submitted with too little memory. Rectified the snakefile to fix this
Initial commit
remove long read output from the file
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The primary focus should be on the files located within pipeline3. The other's were earlier iterations that can be ignored. This pipeline is a major overhaul of the original pipeline. The pipeline use built using snakemake which calls the files in the scripts directory to synthesise the data set. This version removes a lot of the unnecessary writing to disc, and ad hoc data structures and vectorises a lot of the processes that were using too many loops before. The heavy lifting is done by pandas and numpy.
The next steps are to process the data the pipeline has output with CNNs. In addition to this, the current pipeline should have more extensive, and cleaner commenting, along with documentation to ensure that it can be read by future users.