Use MKL in NumPy and SciPy

[AlexKurek]

Description

This builds NumPy and Scipy against MKL. I think it is better this way than #98, since no extra large packages are installed (#98 (comment)).
It does compile. Import numpy and scipy works.
It requires benchmarks. For example, LSMTool and PyBDSF use Numpy and Scipy a lot (https://git.astron.nl/RD/rapthor/-/issues/31).

BTW:

  CPU Optimization Options
    baseline:
      Requested : min+detect
      Enabled   : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
    dispatch:
      Requested : max -xop -fma4
      Enabled   : AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL AVX512_SPR

So it seems that AVX2 is used by default in NumPy and everything "above" is dispatched.

[AlexKurek]

Only test fails after removing numpy and scipy from requirements.txt. The container is built fine.

[tikk3r]

Same as for #231. The test will have to be fixed.

[AlexKurek]

Benchmarks do not look good (i5-11400H):

results_NOmkl

results_mkl

It seems to be slower sometimes. It also seems it is due to the manual build, not to the use of MKL: #240 (comment).
This is the code for Ubuntu 24.10, if you would like to verify or do it for AOCL (I dont have AMD CPU):

### no MKL
python3 -m venv nomkl
source nomkl/bin/activate

pip install numpy==1.26.4
git clone https://github.com/spcl/npbench.git
cd npbench/
python -m pip install -r requirements.txt
python -m pip install .
pip install setuptools
pip install PyQt5
pip install pygount scipy

# https://github.com/spcl/npbench?tab=readme-ov-file#running-benchmarks
python run_framework.py -f numpy

pip install pandas matplotlib
python plot_results.py

deactivate
cd ..


### MKL
python3 -m venv mkl
source mkl/bin/activate

pip install setuptools
pip install PyQt5
pip install pygount

wget https://github.com/numpy/numpy/releases/download/v1.26.4/numpy-1.26.4.tar.gz
tar xzf numpy-1.26.4.tar.gz
rm numpy-1.26.4.tar.gz
cd numpy-1.26.4/
pip install . -Csetup-args=-Dblas-order=mkl,blis -Csetup-args=-Dlapack-order=mkl,lapack
cd ..
rm -rf numpy-1.26.4/
pip show numpy

wget https://github.com/scipy/scipy/releases/download/v1.15.1/scipy-1.15.1.tar.gz
tar xzf scipy-1.15.1.tar.gz
rm scipy-1.15.1.tar.gz
cd scipy-1.15.1/
pip install . -Csetup-args=-Dblas=mkl-dynamic-lp64-iomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-iomp
cd ..
rm -rf scipy-1.15.1/
pip show scipy

cd npbench/
python run_framework.py -f numpy

pip install pandas matplotlib
python plot_results.py

deactivate
cd ..

[tikk3r]

Is it possible to plot the ratio between the two? At least it's not a dramatic difference at a quick visual glance

It's a bit of a shame that it doesn't seem to help, and also a bit unexpected perhaps?

[AlexKurek]

I have some doubts if it gets configured right:
numpy.show_config()

  "Build Dependencies": {
    "blas": {
      "name": "mkl-dynamic-lp64-seq",
      "found": true,
      "version": "2025.0.1",
      "detection method": "pkgconfig",
      "include directory": "/opt/intel//oneapi//mkl/2025.0/lib/pkgconfig/../../include",
      "lib directory": "/opt/intel//oneapi//mkl/2025.0/lib/pkgconfig/../../lib",
      "openblas configuration": "unknown",
      "pc file directory": "/opt/intel/oneapi/mkl/2025.0/lib/pkgconfig"
    },
    "lapack": {
      "name": "dep133228677134864",
      "found": true,
      "version": "1.26.4",
      "detection method": "internal",
      "include directory": "unknown",
      "lib directory": "unknown",
      "openblas configuration": "unknown",
      "pc file directory": "unknown"
    }

Should it be sequential really (mkl-dynamic-lp64-seq)? I manually changed it to mkl-dynamic-lp64-iomp and it seems to be faster than without MKL in many cases now (plot below). So I reopened this PR.
LAPACK seems not detected in numpy. It is detected in Scipy though. Maybe we should construct .pc file for LAPACK in Numpy.
Maybe numpy 1.26.4 is too old for latest MKL.

Is it possible to plot the ratio between the two?

Not in an easy way I think, the output is kind of messy. I attach outputs and commands to do benchmarks.
bench.zip

[tikk3r]

Should it be sequential really (mkl-dynamic-lp64-seq)? I manually changed it to mkl-dynamic-lp64-iomp and it seems to be faster than without MKL in many cases now (plot below). So I reopened this PR.

I think one thing to look out for carefuly here is threading. There are many environment variables that control it and I wouldn't be surprised if by setting the iomp version you're just seeing single threaded vs multithreaded.

[AlexKurek]

LAPACK is detected in numpy 2.2.2:

"lapack": {
  "name": "mkl-dynamic-lp64-iomp",
  "found": true,
  "version": "2025.0.1",
  "detection method": "pkgconfig",
  "include directory": "/opt/intel//oneapi//mkl/2025.0/lib/pkgconfig/../../include",
  "lib directory": "/opt/intel//oneapi//mkl/2025.0/lib/pkgconfig/../../lib",
  "openblas configuration": "unknown",
  "pc file directory": "/opt/intel/oneapi/mkl/2025.0/lib/pkgconfig"
}

So numpy 1.26.4 is too old to detect latest MKL.

[AlexKurek]

There is no benefit from adding b_pgo or b_ndebug (https://mesonbuild.com/Builtin-options.html). So it seems to be good as it is now.

[AlexKurek]

PyBDSF master, Python 3.10, 127M image. But - Numpy 1.26.4 detects only BLAS MKL, not LAPACK MKL. So I suggest to wait with conclusions until Numpy >=2 is well supported.

## no MKL
PyBDSF running time: 0:3:51
PyBDSF running time: 0:3:58

## MKL
PyBDSF running time: 0:3:52
PyBDSF running time: 0:3:57