Module not found: smlp

[Th0ught09]

After running ./smlp_regr.py -w 8 -def n -t all -tol 7, to my understanding, it spawns some threads running src/run_smp.py, as an example in the output

../../src/run_smlp.py -data "../data/smlp_toy_basic.csv" -out_dir ./ -pref Test14 -mode train -resp y1,y2 -feat x1,x2,p1,p2 -model poly_sklearn -save_model_config f -mrmr_pred 0 -plots f -pred_plots f -resp_plots f -seed 10 -log_time f

Running this manually produces an error of

ModuleNotFoundError: No module named 'smlp'

After searching the repo I can't find a file or module called smlp.py

> find smlp.py
find: ‘smlp.py’: No such file or directory

I'd like to try to try run the files individually to test my modifications so any help on this would be appreciated :)

[fbrausse]

I believe you don't have the smlp Python module installed in your PYTHONPATH first, before the regression script or run_smlp.py is executed. The way to install it is described in the README

cd smlp/utils/poly

# [...]

# setup, build & install libsmlp
meson setup -Dkay-prefix=$HOME/kay --prefix $HOME/.local build
ninja -C build install

You may need to adjust the paths to the kay library and to the prefix where libsmlp and the smlp Python module will be installed to.

There is no file smlp.py because the module is defined via the file utils/poly/python/smlp/__init__.py.

(Hint: use find -name __init__.py to search for the file, the first argument to find is a path to start the search at.)

[Th0ught09]

I've been trying to hack away at this but still no luck, I run meson setup -Dkay-prefix=$HOME/kay --prefix $VIRTUAL_ENV build where $VIRTUAL_ENV in the python virtual environment I'm in and I get the output

The Meson build system
Version: 1.9.0
Source dir: /home/kirkm/projects/smlp/smd/utils/poly
Build dir: /home/kirkm/projects/smlp/smd/utils/poly/build
Build type: native build
Project name: poly
Project version: 0.10.1
C compiler for the host machine: gcc (gcc 14.3.0 "gcc (GCC) 14.3.0")
C linker for the host machine: gcc ld.bfd 2.44
C++ compiler for the host machine: g++ (gcc 14.3.0 "g++ (GCC) 14.3.0")
C++ linker for the host machine: g++ ld.bfd 2.44
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: YES (/nix/store/k9s1w95rqfr14lb5xms1lwzrq1yiinzz-pkg-config-wrapper-0.29.2/bin/pkg-config) 0.29.2
Run-time dependency gmp found: YES 6.3.0
Found CMake: /home/kirkm/.nix-profile/bin/cmake (3.31.6)
Run-time dependency z3 found: NO (tried pkgconfig and cmake)
Has header "flint/fmpz.h" : NO
Run-time dependency gmpxx found: YES 6.3.0
Check usable header "gmpxx.h" : YES
Message: using gmpxx for Z and Q
h5c++ found: NO
Run-time dependency HDF5 for cpp found: NO (tried pkgconfig, pkgconfig and config-tool)
Has header "hdf5.h" : NO
h5c++ found: NO
Run-time dependency HDF5 for cpp found: NO (tried pkgconfig, pkgconfig and config-tool)
Has header "H5Cpp.h" : NO
Check usable header "kay/numbers.hh" : YES
Message: have "kay"
Run-time dependency kjson found: NO (tried pkgconfig and cmake)
Has header "kjson.h" : NO
Message: Built-in Z3 support disabled
Message: keras-nn support disabled
Message: external solver support enabled
Message: stand-alone program "smlp" enabled
Configuring config.h using configuration
Program python (fractions) found: YES (/home/kirkm/.conda/bin/python) modules: fractions
Run-time dependency python found: YES 3.12
Run-time dependency Boost found: NO (tried system)
Message: SMLP Python API disabled
Build targets in project: 3

poly 0.10.1

  Dependencies
    gmp                 : YES 6.3.0
    z3                  : NO
    flint               : NO
    gmpxx               : YES 6.3.0
    hdf5                : NO, NO
    kay                 : YES
    kjson               : NO
    python3             : YES 3.12
    boost               : NO

  Build with features
    Interval solver     : NO, z3
    NN                  : NO, kjson, hdf5
    External solver     : YES
    Program "smlp"      : YES
    Python module "smlp": NO, python3, boost

  User defined options
    kay-prefix          : /home/kirkm/kay
    prefix              : /home/kirkm/projects/smlp/smd/.venv

Found ninja-1.13.1 at /nix/store/kkk2nf522g9yi9xk7848sav0ji2jzhcv-ninja-1.13.1/bin/ninja
(base)

When running ninja there's a lot of format warnings and then

Installing src/smlp to /home/kirkm/projects/smlp/smd/.venv/bin
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/common.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/expr.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/prefix.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/infix.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/expr2.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/domain.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/dump-smt2.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/poly.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/include/es/sexpr.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/include/es/sexpr-detail.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/include/es/smtlib2-parser.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/solver.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/ival-solver.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/reals.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/algebraics.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/src/ext-solver.hh to /home/kirkm/projects/smlp/smd/.venv/include/smlp
Installing /home/kirkm/projects/smlp/smd/utils/poly/build/src/config.h to /home/kirkm/projects/smlp/smd/.venv/include/smlp

running a case gives me

python3 src/run_smlp.py -data "../data/smlp_toy_num_resp_mult"       -out_dir ./ -pref Test83 -mode optimize -pareto t       -resp y1,y2 -feat x,p1,p2 -model dt_sklearn -dt_sklearn_max_depth 15       -spec smlp_toy_num_resp_mult_free_inps -data_scaler min_max       -beta "y1>7 and y2>6" -objv_names obj1,objv2,objv3       -objv_exprs "(y1+y2)/2;y1/2-y2;y2" -epsilon 0.05 -delta_rel 0.01       -save_model_config f -mrmr_pred 0 -plots f -seed 10 -log_time f       -spec ../specs/smlp_toy_num_resp_mult_free_inps.spec

2025-11-24 12:09:21.753310: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-11-24 12:09:21.828956: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-24 12:09:24.066275: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
Traceback (most recent call last):
  File "/home/kirkm/projects/smlp/smd/src/run_smlp.py", line 11, in <module>
    from smlp_py.smlp_flows import SmlpFlows
  File "/home/kirkm/projects/smlp/smd/src/smlp_py/smlp_flows.py", line 7, in <module>
    from smlp_py.smlp_models import SmlpModels
  File "/home/kirkm/projects/smlp/smd/src/smlp_py/smlp_models.py", line 20, in <module>
    from smlp_py.train_caret import ModelCaret
  File "/home/kirkm/projects/smlp/smd/src/smlp_py/train_caret.py", line 13, in <module>
    from smlp_py.smlp_terms import TreeTerms
  File "/home/kirkm/projects/smlp/smd/src/smlp_py/smlp_terms.py", line 13, in <module>
    import smlp
ModuleNotFoundError: No module named 'smlp'

and so no luck, the environment is correctly activated to my knowledge

which python3
/home/kirkm/projects/smlp/smd/.venv/bin/python3

[fbrausse]

These are the relevant lines from the meson setup stage:

Program python (fractions) found: YES (/home/kirkm/.conda/bin/python) modules: fractions
Run-time dependency python found: YES 3.12
Run-time dependency Boost found: NO (tried system)
Message: SMLP Python API disabled

First, I'm relatively certain the path to the python binary shouldn't be /home/kirkm/.conda/bin/python but pointing to your venv.

Second, and more importantly, you'll need the boost library. On Ubuntu the package is called libboost-python-dev, on Gentoo it's just "boost" with the relevant USE=python flag. Not sure how it's called on NixOS.

With these issues fixed, it should work. Once it does, you can just try python3 -c 'import smlp' in your venv, no need to run the regression script to test it.

[fbrausse]

PS: For boost you can use meson -Dboost-prefix=PATH/TO/BOOST, for fixing the auto-detected python binary, I'm not sure. Are you running meson with the venv activated?

[Th0ught09]

Sorry for the late reply, yes I am. I figured out the issue being with the way meson is packaged in nixos means it could not resolve the boost dependency. Commenting out those lines in src/meson.build builds successfully and I can now run files, for any future users of nixos using this project this is the flake that ended up working

{
  description = "Shell for smlp";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs  }: {
    devShells.x86_64-linux.default = let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;

    in pkgs.mkShell {
      nativeBuildInputs = [
        pkgs.pkg-config
        pkgs.ninja
        pkgs.cmake          
        pkgs.python311      
      ];

      buildInputs = [
        pkgs.python311Packages.boost
        pkgs.gmp
        pkgs.gmpxx
      ];

      shellHook = ''
        export BOOST_ROOT=${pkgs.lib.getDev pkgs.python311Packages.boost}
        export BOOST_INCLUDEDIR=${pkgs.lib.getDev pkgs.python311Packages.boost}/include
        export BOOST_LIBRARYDIR=${pkgs.lib.getDev pkgs.python311Packages.boost}/lib

        echo "Meson ${pkgs.meson.version}, Boost ${pkgs.python311Packages.boost.version}"

        echo "location being ${pkgs.lib.getDev pkgs.python311Packages.boost}"

        rm -rf build
        meson setup -Dkay-prefix=$HOME/kay --prefix $VIRTUAL_ENV -Dboost-prefix=${pkgs.lib.getDev pkgs.python311Packages.boost} build
        ninja -C build install
      '';
    };
  };
}

running nix develop in utils/poly will build once you've edited src/meson.build in the same directory

[Th0ught09]

Now running into issues with the module itself, the install looks to be successful

The Meson build system
Version: 1.9.1
Source dir: /home/kirkm/projects/smlp/smd/utils/poly
Build dir: /home/kirkm/projects/smlp/smd/utils/poly/build
Build type: native build
Project name: poly
Project version: 0.10.1
C compiler for the host machine: gcc (gcc 14.3.0 "gcc (GCC) 14.3.0")
C linker for the host machine: gcc ld.bfd 2.44
C++ compiler for the host machine: g++ (gcc 14.3.0 "g++ (GCC) 14.3.0")
C++ linker for the host machine: g++ ld.bfd 2.44
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: YES (/nix/store/8vdiwpbh0g4avsd6x5v4s0di32vcl3dp-pkg-config-wrapper-0.29.2/bin/pkg-config) 0.29.2
Run-time dependency gmp found: YES 6.3.0
Run-time dependency z3 found: YES 4.15.4.0
Has header "flint/fmpz.h" : YES
Has header "flint/fmpq.h" : YES
Library flint found: YES
Run-time dependency mpfr found: YES 4.2.2
Message: using Flint for Z and Q
Run-time dependency HDF5 for cpp found: YES 1.14.6
Message: have "hdf5"
Check usable header "kay/numbers.hh" : YES
Message: have "kay"
Has header "kjson.h" : YES
Has header "kjson.hh" : YES
Library kjson found: YES
Message: have "kjson"
Message: Built-in Z3 support enabled
Message: keras-nn support enabled
Message: external solver support enabled
Message: stand-alone program "smlp" enabled
Configuring config.h using configuration
Program python (fractions) found: YES (/nix/store/00x3abm7y8j13i6n4sahvbar99irkc7d-python3-3.11.14/bin/python) modules: fractions
Run-time dependency python found: YES 3.11
Message: 3.11
Message: python311
Run-time dependency Boost (found: python311) found: YES 1.87.0
Message: SMLP Python API enabled
Configuring setup.py using configuration
Build targets in project: 4

poly 0.10.1

  Dependencies
    gmp                 : YES 6.3.0
    z3                  : YES 4.15.4.0
    flint               : YES
    mpfr                : YES 4.2.2
    hdf5                : YES 1.14.6
    kay                 : YES
    kjson               : YES
    python3             : YES 3.11
    boost               : YES 1.87.0

  Build with features
    Interval solver     : YES
    NN                  : YES
    External solver     : YES
    Program "smlp"      : YES
    Python module "smlp": YES

  User defined options
    boost-prefix        : /nix/store/z901ai1xspcxf1qkmlhla9az7ql7ngz2-boost-1.87.0-dev
    kay-prefix          : /home/kirkm/kay
    kjson-prefix        : /home/kirkm/kjson
    prefix              : /home/kirkm/projects/smlp/smd/.venv

But importing smlp leads it to having no modules, causing further errors

python3
Python 3.11.12 (main, Apr  9 2025, 04:04:00) [Clang 20.1.0 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import smlp
>>> dir(smlp)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

If you're unsure here, I'll probably just install an ubuntu partition as this has taken a lot of time, and I'm a bit behind on where I should be

[fbrausse]

I suspect that you're running python3 inside of the src directory of this repo, where -- unfortunately -- import smlp does succeed even if the module is not found (due to the subdirectory of the same name), however, if the (installed) module from utils/poly was found in the PYTHONPATH envvar (or the venv), that shouldn't happen. You can confirm the path of the imported module via print(smlp) in Python.

[Th0ught09]

This is the output from the root of the project (I've cloned the repo into a directory called smd due to my workflow)

[kirkm@nixos:~/projects/smlp/smd]$ python3
Python 3.11.12 (main, Apr  9 2025, 04:04:00) [Clang 20.1.0 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import smlp
>>> print(smlp)
<module 'smlp' from '/home/kirkm/projects/smlp/smd/.venv/lib/python3.11/site-packages/smlp/__init__.py'>
>>> dir(smlp)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>>

[Th0ught09]

Solved this, the __init__.py file was overwritten, I reset the file

[fbrausse]

Solved this, the __init__.py file was overwritten, I reset the file

Great! But do you know how that happened? Is there anything we can do to prevent that?

[mdmitry1]

IMHO having one step clean build should be helpful.
With regards to the root cause of this issue, I had similar problem.
Possible root cause:

head -1 smlp_regr.py
#!/usr/bin/env python3

From above one can see that if python3 is defined as python3.12, it won't work.
My workaround is to add the following commands to the beginning of the run script:

\ln -sf /usr/bin/python3.11 python3
if($PATH !~ \.*) then
    setenv PATH ".:${PATH}"
endif

and the following command at the end of the run script:

\rm -f python3

[Th0ught09]

Solved this, the __init__.py file was overwritten, I reset the file

Great! But do you know how that happened? Is there anything we can do to prevent that?

So sorry lost track of this issue, it must've been me not understanding shell commands and accidentally deleting it. Upon redoing this process I didn't have the same issue :)