Conda/Bifrost 1.0.3 illegal instruction error

[ptrebert]

Hi,
just fyi, after installing Bifrost via Conda/bioconda, it immediately exits with an "illegal instruction" error. Building from source on the same server is possible, though. The machine is equipped with the following CPU model:

model name      : Intel(R) Xeon(R) CPU E7-8857 v2 @ 3.00GHz
stepping        : 7
microcode       : 0x715
[...]
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d

Best,
Peter

[GuillaumeHolley]

Hi @ptrebert,

Thanks for letting me know about this issue. When you installed Bifrost with Conda, did you run it afterwards on the same machine (the one used for the Conda install) or a different one?

Guillaume

[ptrebert]

Hi Guillaume,
yes, all steps were executed on the same machine.

Peter

[GuillaumeHolley]

Hi @ptrebert,

"Illegal instructions" is usually an error occurring when you compile the program on machine A and execute it on machine B. By default, Bifrost compiles with "-march=native" as compile option, meaning it can optimize the code for the architecture it is compiled on. By executing on a different machine, the architecture might be different, hence the "Illegal instructions". Anyway, as you are compiling and executing on the same machine, this doesn't seem to be the issue.

Surely the source of the problem is that Bifrost uses AVX2 instructions for the construction only when AVX2 available ("-march=native"). Normally, using AVX2 instructions should be deactivated for your CPU (I see your CPU is limited to AVX) but for some reasons, it is possible that it is still activated. Hence, Bifrost tries to use AVX2 instructions while you don't have them available. How would that be possible? Honestly, I don't know. I would say "probably a Conda issue" as the source install version worked just fine but I don't want to blame anyone else than me for now :) I will look into it.

[ptrebert]

Hi Guillaume,
thanks for all your feedback, and, yes, I agree, machine incompatibility is the most likely explanation; where this is coming from in cases like this is also unclear to me (I had the same issue already with Racon). So my expectation was clearly not that you fix that right away, but rather that it shouldn't go unnoticed, in case other users run into the same problem - or, of course, somebody finds this issue by chance and has a solution :)
Best,
Peter

[winni2k]

Conda installs precompiled binaries, so I would expect that if you want to distribute a conda package of bifrost without avx2 support, then you need to turn off avx2 support during compilation of the conda package (that is, in the bioconda recipe). I haven't found a package that does this yet, but I would guess numpy or tensorflow would have similar issues.

[GuillaumeHolley]

I should have done that some time ago: I will add an extra option in the Cmake for AVX2: OFF to deactivate, ON to activate or NATIVE to have it ON if AVX2 is available. I'm not sure if there is a way to modify the conda recipe in a way that the end user would be able to choose one of these options when installing. But once again, installing Bifrost from source is rather straightforward so I really think the conda package is a waste of time.

[GuillaumeHolley]

Hey @ptrebert and @winni2k,

I added two options to the cmake file:

• -DENABLE_NATIVE_ARCH=ON/OFF to enable/disable native compilation (-march=native). Default is ON.
• -DENABLE_AVX2=ON/OFF to enable/disable AVX2 instructions. Default is ON.

Those options are now explained in the README file and I also updated the Troubleshooting section with the Illegal instruction error.

[winni2k]

I found a user question on SO about AVX2, etc. being disabled in the pip version of tensorflow.

I think one workaround would be to create two packages: one with native compilation disabled, and one with lots of extensions enabled. In the mean-time, I think it's probably fine just to push the version without native extensions to bioconda so that users have a stable version that they can easily install and use.

[GuillaumeHolley]

The main pro of Bioconda is in my opinion that it makes the install easy. How easy in this case? Well, not much easier than the source install (which has no dependency). Based on this, I'm not sure I want to disable native compilation and AVX2 instructions, which bring a nice speed up to the software, for an install which is going to be a little bit easier. Another thing that bothers me here is that large k-mers (k>31) is a compile time variable that cannot be changed with Bioconda, so anyone that works with k>31 will have to use the source install anyway.

[winni2k]

I can see where you're coming from. If it's easy to compile, isn't that as easy to install as a conda package? I don't think that's the case, and there's some research by Serghei Manghul that backs me up on this. However, you don't need to worry about conda packaging if you don't want to. Bifrost is available on a permissive licence, and sooner or later someone who cares enough will fix the bioconda recipe for the bioinformatics community.

[winni2k]

I did a bit more reading on the state of native support in conda. On bioconda-recipes there's an open thread on the subject, and a few more closed ones. It appears that native compilation has been an on-going subject of discussion in the conda community, and conda has so far defaulted to wide-ranging compatibility by only assuming an x86-64 architecture.

There is something called build variants in conda, and there's this package that appears to provide a path for supplying multiple binaries for different cpu architectures: https://github.com/jjhelmus/selectable_pkg_variants_example

[winni2k]

Regarding the compile-time kmer size definition, this issue exists in McCortex as well. There, the solution is to built a kmer specific binary for a kmer size of interest (e.g. mccortex31). There is a shell script included with mccortex that selects the correct binary at run time (i.e. 'mccortex <kmer_size> [mccortex_args...]`).

For conda packaging, the bioconda recipe simply compiles a few binaries for a range of kmer sizes and includes them in the package. That seems to take care of most people's (i.e. my) requirements.

[GuillaumeHolley]

Hi @winni2k,

Thanks for all these information. I get your point regarding bioconda and the community fixing recipes. Yet, I feel very responsible for these recipes. If I just keep updating the source and no one update the recipes, people will end up using deprecated or buggy versions of Bifrost. So it's not that I don't want to take care of it, I really do but I'm lacking the time for it. But hey, no one said that maintaining software was going to be easy :)

Thank you for your link on the bioconda meta-packages, it seems to be the best solution at the moment for the problem at hand and I will probably do something like that for the next recipe update.

Thanks for the tip with McCortex. There is also BCALM2 which you can precompile for a range of k-mer sizes at first and the software will take care at run time of picking the right data structures/libraries. The thing is that for Bifrost, I would like some solution I could include directly in the API. It feels that it would be possible with a major refactoring. The idea (maybe naive) is that the MAX_KMER_SIZE parameter is mostly used in the type Kmer as a static variable. Type Kmer is currently exposed to the user in the API (some functions take as argument or return objects of type Kmer). By taking it or return strings/char arrays instead, this type would be used only internally by Bifrost and there, I would have the freedom of having something more complicated where the k-mer size would not be a static parameter anymore. It would be simpler for the user, no additional compile parameter, but the code of Bifrost would be more complex.

[ptrebert]

Though I don't have anything to add in particular about Conda recipes and related build issues, I leave the below code snippets here for people who want to automatically install Bifrost on various infrastructure potentially supporting the AVX2 extension (and cannot wait for appropriate builds being available in a Conda channel 😉 ).

I use the following Conda environment (simpler specifications would likely work as well) as part of a Snakemake pipeline, and the setup works just fine inside this environment:

name: compile
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.6
  - pip=20.1.1
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_llvm
  - binutils-meta=1.0.4=0
  - binutils_impl_linux-64=2.34=h53a641e_2
  - binutils_linux-64=2.34=hc952b39_20
  - c-compiler=1.0.4=h516909a_0
  - cmake=3.17.0=h28c56e5_0
  - compilers=1.0.4=0
  - cxx-compiler=1.0.4=hc9558a2_0
  - fortran-compiler=1.0.4=he991be0_0
  - gcc_impl_linux-64=7.5.0=hd420e75_6
  - gcc_linux-64=7.5.0=h09487f9_20
  - gfortran_impl_linux-64=7.5.0=hdf63c60_6
  - gfortran_linux-64=7.5.0=h09487f9_20
  - glib=2.64.3=h6f030ca_0
  - gxx_impl_linux-64=7.5.0=hdf63c60_6
  - gxx_linux-64=7.5.0=h09487f9_20
  - ld_impl_linux-64=2.34=h53a641e_2
  - libblas=3.8.0=16_openblas
  - libcblas=3.8.0=16_openblas
  - libclang=9.0.1=default_hde54327_0
  - libcurl=7.69.1=hf7181ac_0
  - libgcc-ng=9.2.0=h24d8f2e_2
  - libgfortran-ng=7.5.0=hdf63c60_6
  - liblapack=3.8.0=16_openblas
  - libllvm9=9.0.1=he513fc3_1
  - libopenblas=0.3.9=h5ec1e0e_0
  - libstdcxx-ng=9.2.0=hdf63c60_2
  - llvm-openmp=10.0.0=hc9558a2_0
  - pkg-config=0.29.2=h516909a_1006
  - pthread-stubs=0.4=h14c3975_1001
  - xz=5.2.5=h516909a_0
  - zlib=1.2.11=h516909a_1006

Just following the setup instructions and running cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX .. && make && make install installs the Bifrost binary into the Conda environment, i.e., Bifrost is available in $PATH after activating the Conda environment.

[winni2k]

This is really cool @ptrebert! Thanks for sharing! I might start doing this kind of thing in some of my snakemake rules with other tools that would benefit from native compilation.