Low model quality for Intel s2_tx_anonym test

[mdmitry1]

Reproduction:

1. Run commands:

\rm -rf /tmp/smlp_issue_12272025 >& /dev/null
mkdir /tmp/smlp_issue_12272025
cd /tmp/smlp_issue_12272025 
git clone ssh://git@github.com/SMLP-Systems/smlp 
cd smlp
git switch SMLP_TUTORIAL 
cd tutorial/examples/s2_tx_anonym/smlp
./run_s2_tx_anonym

2. Wait until file S2_TX_ANONYM_S2_TX_ANONYM_labeled_prediction_precisions.csv will be created
3. Run command:

cat S2_TX_ANONYM_S2_TX_ANONYM_labeled_prediction_precisions.csv

4. Result, which I'm getting:

cat S2_TX_ANONYM_S2_TX_ANONYM_labeled_prediction_precisions.csv
response,msqe,r2_score
o0,42.03761419143268,0.37793910868538405
o1,840.9748218074963,0.09771289275502404

[mdmitry1]

The problem seems to be with input data.
I've tried random forest model, advised by Claude and got similar results.
Reproduction:

\rm -rf /tmp/smlp_issue_12272025 >& /dev/null
mkdir /tmp/smlp_issue_12272025
cd /tmp/smlp_issue_12272025 
git clone ssh://git@github.com/SMLP-Systems/smlp 
cd smlp
git switch SMLP_TUTORIAL 
cd tutorial/examples/s2_tx_anonym/smlp
pytest -s tutorial/examples/s2_tx_anonym

Results:

=================================================================== test session starts ====================================================================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/smlp_issue_12272025/smlp
plugins: anyio-4.12.0, typeguard-4.4.4, mock-3.12.0
collected 1 item                                                                                                                                           

tutorial/examples/s2_tx_anonym/py/test_s2_tx_anonym.py::test_s2_tx_anonym_csv 
Data shape: (2240060, 9)
Columns: ['CH', 'RANK', 'Byte', 'p0', 'p1', 'p2', 'p3', 'o0', 'o1']

First few rows:
   CH  RANK  Byte  p0  p1  p2  p3    o0    o1
0   0     0     0  15   3  60  40 -8.25   0.0
1   0     0     0  15   3  60  40 -8.25   4.0
2   0     0     0  15   3  60  40 -8.25   8.0
3   0     0     0  15   3  60  40 -8.25  12.0
4   0     0     0  15   3  60  40 -8.25  16.0

Number of unique fixed parameter combinations: 32

Training surrogate models...
max_depth=17; n_estimators=18; min_samples_split=20
Model o0 R² score: 0.3885
Model o1 R² score: 0.0994
PASSED

==================================================================== 1 passed in 52.65s ====================================================================

[mdmitry1]

Tried s2_rx_anonym and got similar results.

1. SMLP decision tree:

dt_sklearn_max_depth 15
response,msqe,r2_score
o0,12.640720284770971,0.20418951013871545
o1,1128.8599465331718,0.0961480159072744

2. sklearn random forest:

max_depth=17; n_estimators=18; min_samples_split=20
Model o0 R² score: 0.2315
Model o1 R² score: 0.1007

[zurabksmlp]

I think these datasets were analysed using NNs, because in 2020 when we released these benchmarks (s2_tx/rx and similarly named benchmarks) we only had NN support in SMLP. Please can you try NN? In an earlier email I have sent you some info on NN -related command line options. SMLP toy regression also has many NN tests. Zurab

…

On Mon, Dec 29, 2025 at 2:29 PM Dmitry Messerman ***@***.***> wrote: *mdmitry1* left a comment (SMLP-Systems/smlp#39) <#39 (comment)> Tried *s2_rx_anonym* and got similar results. 1. SMLP decision tree: dt_sklearn_max_depth 15 response,msqe,r2_score o0,12.640720284770971,0.20418951013871545 o1,1128.8599465331718,0.0961480159072744 2. sklearn random forest: max_depth=17; n_estimators=18; min_samples_split=20 Model o0 R² score: 0.2315 Model o1 R² score: 0.1007 — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPWPDL5CUXVXXQTEE2X6LSD4EENDNAVCNFSM6AAAAACQFT3QWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOJWGM4TKOBYHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mdmitry1]

This is what I see in README_bench:

egrep "/s2_rx_anonym.spec|/s2_tx_anonym.spec" $(git rev-parse --show-toplevel)/bench/README_bench | tr ' ' '\n' | grep -A1 model | sort -u -r | egrep -v '^\-\-' 
-model
dt_sklearn

[zurabksmlp]

Yes the README I created maybe 1 year or so ago, These are not necessarily the same commands as ones used in the conference paper. Anyway, you are welcome to try NN instead of tree models.

…

On Mon, Dec 29, 2025 at 4:20 PM Dmitry Messerman ***@***.***> wrote: *mdmitry1* left a comment (SMLP-Systems/smlp#39) <#39 (comment)> This is what I see in README_bench: egrep "/s2_rx_anonym.spec|/s2_tx_anonym.spec" $(git rev-parse --show-toplevel)/bench/README_bench | tr ' ' '\n' | grep -A1 model | sort -u -r | egrep -v '^\-\-' -model dt_sklearn — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPWPDLYPC4EVBKCBOYSRD6L4EE2DNAVCNFSM6AAAAACQFT3QWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOJWGY2TSNZTHE> . You are receiving this because you commented.Message ID: ***@***.***>

[mdmitry1]

I tried neural network from sklearn. As I have old computer it took much longer, but I've got similar results.
See below for s2_tx_anonym:

Training neural network surrogate models...
Number of hidden layers: 2; number of neurons=40
Hidden layers size: (40, 40); max_iter=20
Training model for o0...
Training completed. Iterations: 20
Training model for o1...
Training completed. Iterations: 20
Model o0 R² score: 0.3553
Model o1 R² score: 0.0963

As o1 score is consistently lower that for o0, I believe the issue is in my understanding of the data or data itself.

[zurabksmlp]

Please try -split 1. This option instructs SMLP to use the entire dataset for training. This usually is not a good idea since models can overfit, but for optimization use cases we often (or essentially always) do that because data is so valuable.

…

On Mon, Dec 29, 2025 at 5:37 PM Dmitry Messerman ***@***.***> wrote: *mdmitry1* left a comment (SMLP-Systems/smlp#39) <#39 (comment)> I tried neural network from sklearn. As I have old computer it took much longer, but I've got similar results. See below for s2_tx_anonym: Training neural network surrogate models... Number of hidden layers: 2; number of neurons=40 Hidden layers size: (40, 40); max_iter=20 Training model for o0... Training completed. Iterations: 20 Training model for o1... Training completed. Iterations: 20 Model o0 R² score: 0.3553 Model o1 R² score: 0.0963 As o1 score is consistently lower that for o1, I believe the issue is in my understanding of the data or data itself. — Reply to this email directly, view it on GitHub <#39 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPWPDL7MKZQFUJBANI7GPXL4EFDDVAVCNFSM6AAAAACQFT3QWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOJWHA2DCMRYGU> . You are receiving this because you commented.Message ID: ***@***.***>