Error exsecute HotNet2

[adj9]

Good morning,
I tried executing HotNet2 on a Mac with Python 3.6.5 (command: python -V). As recommended on the README.txt file (present in the project’s root), I used virtualenv (v. 16.2.0, installed with the command: pip install virtualenv).

I used the “paper_commands.sh” file to see the commands to be executed via terminal. I based myself on:
> python ../makeNetworkFiles.py -e  data/networks/hint+hi2012/hint+hi2012_edge_list -i  data/networks/hint+hi2012/hint+hi2012_index_gene -nn hint+hi2012 -p  hint+hi2012 -b  0.4 -o  data/networks/hint+hi2012 -np 100 -c  1

Executing it, it gives me this error:
File "makeNetworkFiles.py", line 60
if not args.only_permutations:
^
SyntaxError: invalid syntax

The error is related to the -op (only permutation) parameter which is not defined in the command. So, I did some tests adding the file “data/heats/pan12.gene2freq.txt”, present in the project, but I still do not work (so it becomes python ../makeNetworkFiles.py -e  data/networks/hint+hi2012/hint+hi2012_edge_list -i  data/networks/hint+hi2012/hint+hi2012_index_gene -nn hint+hi2012 -p  hint+hi2012 -b  0.4 -o  data/networks/hint+hi2012 -np 100 -c  1 -op data/heats/pan12.gene2freq.txt).

Later, I went down to the "example" folder. By consulting the README.txt file in the folder I tried to run the command:
> python makeRequiredPPRFiles.py @example/configs/influence_matrix.config

Even though it was executed, it still couldn’t find the file “makeRequiredPPRFiles.py”, giving me the error:

python: can't open file 'makeRequiredPPRFiles.py': [Errno 2] No such file or directory

Do you have any suggestion?

Best regards

Alessandro LUMACA

[matthewreyna]

Thanks for your questions. Could you try the paper_commands.sh script again using Python 2 to see if the error in the makeNetworkFiles.py script disappears?

We replaced the makeRequiredPPRFiles.py script with the makeNetworkFiles.py script to make HotNet2 easier to use, so previous examples will not work.

[adj9]

I tried to create a virtualenv with python 2.7.10 with the command:
> virtualenv --python=<path_virtualenv> <path_python>

I execute the virtual machine with the command:
source venv/bin/activate

and after I install che requirements with the command:
> pip install -r requirements.txt

Execute the first command of paper_commands.sh file that is:
(venv) > python makeNetworkFiles.py -e data/networks/hint+hi2012/hint+hi2012_edge_list -i data/networks/hint+hi2012/hint+hi2012_index_gene -nn hint+hi2012 -p hint+hi2012 -b 0.4 -o data/networks/hint+hi2012 -np 100 -c 1

the error persists

Moreover, I tried to run the command:
(venv) > python makeHeatFile.py scores -hf data/heats/pan12.gene2freq.txt -o data/heats/pan12.gene2freq.json -n pan12.freq

and the error is the following:
Traceback (most recent call last):
File "makeHeatFile.py", line 4, in
from hotnet2 import heat as hnheat, hnap, hnio
File "~/hotnet2-master/hotnet2/init.py", line 1, in
from constants import *
ModuleNotFoundError: No module named 'constants'

Do you have any suggestion?

Best regards

Alessandro LUMACA

[matthewreyna]

I would double check to make sure that you are actually running Python 2 and not Python 3 here. The ModuleNotFoundError is a common Python 3 error because Python 2 and 3 handle imports (relative vs. absolute) differently. Please see this post:
https://stackoverflow.com/questions/43728431/relative-imports-modulenotfounderror-no-module-named-x

If the error persists, then there may be another issue here.

[adj9]

Thank you for the answer. I tried running HotNet2 on two virtualenv: one with Python 2 and the other with Python 3.

I stopped using virtualenv because I have problems with the h5py package on the Mac (macOS Mojave, v: 10.14.2, 64bit). I install it with pip but I have some troubles when it comes to importing it.

The based command is:
(venv) > python makeHeatFile.py scores -hf data/heats/pan12.gene2freq.txt -o data/heats/pan12.gene2freq.json -n pan12.freq

I made some transformations of the print commands and some import, from Python 2 to Python 3.
But I stopped at the following error:
WARNING: Could not import C module; falling back to NumPy for similarity matrix creation.
Traceback (most recent call last):
File "makeHeatFile.py", line 4, in
from hotnet2 import heat as hnheat, hnap, hnio
File “/hotnet2/init.py", line 3, in
from . import delta
File “/hotnet2/delta.py", line 9, in
from . import hotnet2 as hn, hnio
File “~/hotnet2/hnio.py", line 8, in
from . import component_sizes
ImportError: cannot import name 'component_sizes'

I stopped at this error, it can not import "component_sizes" from the file “~/hotnet2/hnio.py" and I can not find the respective * .py file in the project.

In case some passages are not clear, ask me as well.

Best regards
Alessandro LUMACA

[matthewreyna]

It looks like you are changing parts of the code to run with Python 3, but the updated code still has errors in Python 3. I would recommend running HotNet2 without changing the code with Python 2, which we have used successfully on a number of systems, including macOS.

You may also want to try Hierarchical HotNet, which has many improvements over HotNet2 including full Python 2/3 compatibility. Here are links to the to the paper and GitHub repository:
https://academic.oup.com/bioinformatics/article/34/17/i972/5093236
https://github.com/raphael-group/hierarchical-hotnet

[adj9]

Good morning
I'm trying to run HotNet2 with Python 2.7.15 (command python --version, not virtualenv as before).
I started from the script I found in the folder "paper" and run it with the terminal command:
> sh paper_commands.sh

The script *.sh executes it, but not completely. Running the first script (makeNetworkFiles.py) gives me the following warning:

WARNING: Could not import C module; falling back to NumPy for similarity matrix creation.
WARNING: Output directory is not empty. Any conflicting files will be overwritten. (Ctrl-c to cancel).

The NumPy library (v. 1.16.0) was previously installed via pip 9.0.3 (command: pip install numpy).

The execution of the * .sh script, however, continues and generates the *.json files in the folder "paper/data/heats" and the *.h5 files in the folder ”data/networks/xxx".

After, with the execution of the script HotNet2.py, I saw that it goes to the files that are in the folder "data/networks/xxx/permuted" (option: -pnp) but the folder remains empty.

Also, running the paper_commands.sh script goes a term considered in the "scripts/permuteNetwork.py" file I commented on the instruction
https://github.com/raphael-group/hotnet2/blob/0f12ff0112ee00033f4163666cef665c05bb4c6e/scripts/permuteNetwork.py#L73
swaps = pool.map (permute_network_wrapper, jobArgs)

and related ones:
https://github.com/raphael-group/hotnet2/blob/0f12ff0112ee00033f4163666cef665c05bb4c6e/scripts/permuteNetwork.py#L83
https://github.com/raphael-group/hotnet2/blob/0f12ff0112ee00033f4163666cef665c05bb4c6e/scripts/permuteNetwork.py#L84

Otherwise with debugging I saw that the program stays on that line of code. I have a MacBook with a i7 of the first generation.

Do you have any suggestion?

Best regards

Alessandro

[matthewreyna]

The makeNetworkFiles.py script warnings are not important warnings, so you can ignore them.

From your email, it is not clear to me whether or not you changed the scripts/permuteNetwork.py script and whether or not it finishes without an error message. The scripts/permuteNetwork.py script can take several hours or days to run, especially on a laptop like your MacBook, so I would recommend running it on another machine if possible. For the sake of debugging, you can try reducing the number of network permutations at the top of the paper_commands.sh script to a much smaller number, such as num_network_permutations=4, to see if there is an actual error or if the script is taking more time than you expect to finish.

[adj9]

It is true, it is not very clear. I'll explain to you better what I did.

I have commented only lines 73, 83 and 84 of file "scripts/permuteNetwork.py", since with the debugging the program is looped to line 73.

By running all the "scripts/permuteNetwork.py" python codes from the script paper_commands.sh, the folders "~/permuted/" are empty.

For every command that executes, the program gives a warning because it does not find the file in the folder "~/permuted/".
An example of a warning is:
IOError: [Errno 2] No file or directory of this type: 'data/networks/hint+hi2012/permuted/hint+hi2012_edgelist_1'

Since you tell me that execution can take many hours, I try to decrease the permutations of the network and check for any errors.

Best regards

Alessandro

[matthewreyna]

If you comment-out these lines of this code, then HotNet2 will not generate permuted networks, which is why these folders are empty and why you are receiving errors. I recommend running the code without changing it (you may want to clone it from GitHub again) using many fewer permutations to see if the issue persists.

[adj9]

Good morning,
I have solved the problems that I had before, decreasing the number of permutations of the reference network, I get *.h5 files and the two *.json files to describe the genes. Now I have to run HotNet2 and from error in writing its results (parameter -o).
The error is:

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
Traceback (most recent call last):
  File "../HotNet2.py", line 139, in <module>
    run(get_parser().parse_args(sys.argv[1:]))
  File "../HotNet2.py", line 85, in run
    single_runs, consensus, linkers, auto_deltas, consensus_stats = consensus_with_stats(args, networks, heats)
  File "~/HotNet2/hotnet2/consensus.py", line 17, in consensus_with_stats
    single_runs, consensus, linkers, auto_deltas = consensus_run( args, networks, heats, verbose )
  File "~/HotNet2/hotnet2/consensus.py", line 68, in consensus_run
    heat, addtl_genes = filter_heat(heat, None, False, 'There are ## genes with heat score 0')
  File "~/HotNet2/hotnet2/heat.py", line 23, in filter_heat
    min_score = min([score for score in heat.values() if score > 0])
ValueError: min() arg is an empty sequence

The script ends and does not generate the "result" folder (passed with the -o parameter of the python script HotNet2.py).
Searching on the Internet I have the same error described here: https://github.com/pierluigiferrari/ssd_keras/issues/213 where is used Python 3.6, but says that the error is deprecated.

I tried to execute the code with both Python 2.7.15 and version 3.6 and the error persists.

Do you have any suggestion?

Thank,
Best regards

Alessandro

[matthewreyna]

You are receiving this error because none of the genes in the largest connected component of your network has a positive heat score. Please see the data from the HotNet2 paper for examples:
https://github.com/raphael-group/hotnet2/tree/master/paper/data