Cython v3 support #426

munechika-koyo · 2023-07-28T10:00:45Z

Hello,

I tried to resolve cython v3 incompatibilities by adding reversed binary arithmetic operators like __radd__, __rmul__, etc.
Existing unit test has been passed, but I did not make sure that they cover all of methods.
I would appreciate it if you would check & comment on this PR.

vsnever

Hi @munechika-koyo,

I have looked at the code and I think the new binary methods are implemented correctly. Plus, all the tests and demos I've run are working fine.

Cython currently generates tons of warnings about deprecated DEF statements. However, I think that so far there is no adequate replacement for them. Although many of these statements can be replaced with global cdef variables, this does not work for those that define the size of static arrays, like here:

source/raysect/core/math/random.pyx

Lines 99 to 104 in ea5bc96

    
           DEF NN = 312 
        
           DEF MM = 156 
        
           # The array for the state vector 
        
           cdef uint64_t mt[NN]

There is a draft PR with a proposal to add global constants (e.g. cdef const double CONST), cython/cython#5242. I think that until this or similar functionality is implemented, DEF statements in Raysect or Cherab should remain.

Also, Cython 3 allows to get rid of that annoying deprecated NumPy-1.7 C-API warning. This can be done by defining the macro in setup.py:

macros = [("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")]
...
                    extensions.append(Extension(module, [pyx_file], include_dirs=compilation_includes, extra_compile_args=compilation_args, define_macros=macros),)
...
                    extensions.append(Extension(module, [c_file], include_dirs=compilation_includes, extra_compile_args=compilation_args, define_macros=macros),)

munechika-koyo · 2023-08-02T07:42:28Z

Thank you for confirming my PR! @vsnever

There is a draft PR with a proposal to add global constants (e.g. cdef const double CONST), cython/cython#5242. I think that until this or similar functionality is implemented, DEF statements in Raysect or Cherab should remain.

Yes, I agree with you. We should follow this issue and check at which version DEF statement will be depricated.

Also, Cython 3 allows to get rid of that annoying deprecated NumPy-1.7 C-API warning. This can be done by defining the macro in setup.py:

I added some commits according to your recommendation. Please confirm it as well.

vsnever

Thanks for this, @munechika-koyo.
Please note that my approval only means that I agree to these changes as a user. Only @CnlPepper decides what changes will be in the Raysect code.

jacklovell · 2023-10-20T09:09:57Z

Cython currently generates tons of warnings about deprecated DEF statements. However, I think that so far there is no adequate replacement for them. Although many of these statements can be replaced with global cdef variables, this does not work for those that define the size of static arrays, like here:

It is possible to have compile time constant integers in Cython without DEF, by using an anonymous enum. Your example would become:

cdef enum:
    NN = 312 
    MM = 156 
  
 # The array for the state vector 
 cdef uint64_t mt[NN]

Is it clearer than a bunch of DEF statements? I'm not convinced. Is it nicer than being spammed with warnings about DEF being deprecated? Probably.

It would at least solve this specific case of defining the size of statically-sized arrays, as well as a bunch of other integer definitions in the code. The DEF INFINITY instances can be replaced by from math cimport INFINITY. There are also a lot of places in the code where DEF is used for floats which will need a different solution: none of them quite fit as math cimports.

vsnever · 2023-10-20T21:36:56Z

It would at least solve this specific case of defining the size of statically-sized arrays, as well as a bunch of other integer definitions in the code. The DEF INFINITY instances can be replaced by from math cimport INFINITY. There are also a lot of places in the code where DEF is used for floats which will need a different solution: none of them quite fit as math cimports.

It looks like the Cython developers have removed the deprecation warnings for DEF statements for now, cython/cython#4310 (comment). While switching to enum for integers is ok, I'd wait to see if the Cython developers come up with solution for compile-time constants that replaces DEF.

jacklovell

This generally looks OK to me. As well as a few inline comments, I'd suggest the following:

Please update pyproject.toml to specify cython~=3.0 in the build requirements.
Please remove emojis from your commit messages: they don't render properly in the pager git uses by default on many OSs.

Tests all pass without issue, but there are some build warnings when building with Cython 3.0.11. There are lots of "performance hints" such as:

performance hint: raysect/core/math/cython/interpolation/linear.pxd:38:20: No exception value declared for 'linear2d' in pxd file.
Users cimporting this function and calling it without the gil will always require an exception check.
Suggest adding an explicit exception value.

And

performance hint: raysect/core/math/statsarray.pyx:707:5: Exception check on '_add_sample' will always require the GIL to be acquired.
Possible solutions:
        1. Declare '_add_sample' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exce
ptions.
        2. Use an 'int' return type on '_add_sample' to allow an error code to be returned.

Personally I don't think these are a big issue at the moment, since Raysect is used with multiprocessing rather than multithreading, and cleaning them all up would drastically increase the scope of this PR. But maybe worth addressing in a subsequent PR, especially if the Python 3.13 free-threading build is going to be getting more attention in future.

There's also a number of NumPy API warnings, e.g.:

raysect/optical/spectrum.c: In function ‘__pyx_f_7raysect_7optical_8spectrum_8Spectrum__construct’:
raysect/optical/spectrum.c:21871:3: warning: passing argument 1 of ‘PyArray_DATA’ from incompatible pointer type [enabled by default]
   PyArray_FILLWBYTE(__pyx_t_1, 0);
   ^
In file included from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:0,
                 from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from raysect/optical/spectrum.c:1259:
/home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1470:1: note: expected ‘struct PyArrayObject *’ but argument is of type ‘struct PyObject *’
 PyArray_DATA(PyArrayObject *arr)
 ^
raysect/optical/spectrum.c:21871:3: warning: passing argument 1 of ‘PyArray_ITEMSIZE’ from incompatible pointer type [enabled by default]
   PyArray_FILLWBYTE(__pyx_t_1, 0);
   ^
In file included from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:0,
                 from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from raysect/optical/spectrum.c:1259:
/home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1524:1: note: expected ‘const struct PyArrayObject *’ but argument is of type ‘struct PyObject *’
 PyArray_ITEMSIZE(const PyArrayObject *arr)
 ^
raysect/optical/spectrum.c:21871:3: warning: passing argument 1 of ‘PyArray_DIMS’ from incompatible pointer type [enabled by default]
   PyArray_FILLWBYTE(__pyx_t_1, 0);
   ^
In file included from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:0,
                 from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from raysect/optical/spectrum.c:1259:
/home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1482:1: note: expected ‘struct PyArrayObject *’ but argument is of type ‘struct PyObject *’
 PyArray_DIMS(PyArrayObject *arr)
 ^
raysect/optical/spectrum.c:21871:3: warning: passing argument 1 of ‘PyArray_NDIM’ from incompatible pointer type [enabled by default]
   PyArray_FILLWBYTE(__pyx_t_1, 0);
   ^
In file included from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:0,
                 from /home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from raysect/optical/spectrum.c:1259:
/home/jlovell/venvs/raysect-dev/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1464:1: note: expected ‘const struct PyArrayObject *’ but argument is of type ‘struct PyObject *’
 PyArray_NDIM(const PyArrayObject *arr)
 ^

It's unclear to me if this is a Cython bug or a Raysect bug: the Raysect code in the pyx file looks OK to me, and the warnings aren't produced when building master (using Cython 0.29.37 and without defining NPY_NO_DEPRECATED_API).

raysect/core/math/point.pyx

jacklovell · 2024-10-28T17:13:54Z

raysect/core/math/vector.pyx


            return NotImplemented

-    # TODO - add 2D affine transformations


Keep this TODO?

If someone does not implement 2D affine matrices, I will remove this comment and replace NotImplemented with throwing the built-in exception, such as TypeError.

setup.py

vsnever · 2024-10-31T08:50:30Z

It's unclear to me if this is a Cython bug or a Raysect bug: the Raysect code in the pyx file looks OK to me, and the warnings aren't produced when building master (using Cython 0.29.37 and without defining NPY_NO_DEPRECATED_API).

Just checked the Cython output, Cython correctly casts pointer types following the definitions in the NumPy array API. It looks like this was a NumPy bug because I no longer get these warnings with NumPy 2.1

This commit works in progress

- mainly function base modules are modified

this modification is derived from the cython 3.0 compile error.

munechika-koyo · 2025-07-17T20:34:43Z

@CnlPepper @jacklovell @vsnever @Mateasek
Because the limitation of cython<3.0 prevents Raysect from being compatible with Python 3.13, I would like to re-activate this discussion and merge this change into the main branch.

As far as I understand, the remaining issue is around the DEF statement, and the workaround would be replacing enum, from math cimport INFINITY, etc, following the v3 migration guide.
I will add this modification later in this PR.

Or, according to cython/cython#4310, DEF is not deprecated as of Cython 3.1.
So, it is an option to keep them in this PR and postpone the revision with other new PR (or issues).

Which do you think is the best choice? Or does anybody have better ideas?

jacklovell · 2025-07-18T12:36:48Z

Having thought about this more, I would suggest following the Cython migration guide and replacing the int DEF statements with anonymous enums and cimporting other available constants. For float DEFs the alternative is I guess cdef const for Cython >=3.1 and hoping the compiler inlines it, though that would mean dropping support for Python <=3.7. Perhaps it's about time for that.

munechika-koyo · 2025-07-18T13:28:11Z

@jacklovell Thank you for your prompt response!
I summarize the TODO list as:

Replace int DEF statements with anonymous enums.
Use cimport for INFINITY (from libc.math cimport HUGE_VAL as INFINITY)
Replace float DEF with cdef const with cython 3.1.*
Pin the cython~=3.1, requires-python = ">= 3.8"

Based on this list, I will add the commits later.

munechika-koyo · 2025-07-18T14:01:03Z

I noticed that INFINITY defined in libc.math was typed as float (const float INFINITY).
To maintain type consistency, should we use HUGE_VAL defined as const double HUGE_VAL?

jacklovell · 2025-07-18T16:06:59Z

INFINITY seems more mathematically relevant here, and would be better understood in the "1/INFINITY = 0" sense for a scientifically-minded reader of the code. HUGE_VAL to me seems more like an implementation detail. But Alex may have had good reason for using 1e999 (another huge val) originally which I've missed.

CnlPepper · 2025-07-18T16:43:42Z

Hi all, the infinity being 1e999 is entirely a legacy thing and should be replaced with the cmath infinity. That definition used to be missing.

DEF's should not be removed until cython have developed an alternative solution. The cython user base (unsurprisingly) reacted poorly to their propsed removal.

I'm going to spend aome time this weekend and next moving raysect (and some other projects) to cython 3 and meson-python (the auto compilation is so useful). I'm also working on a patch to cython to fully overhaul the pxd/module discovery to handle meta finders. This should hopefully, one and for all, fix isaues with cython being unable to find pxd file in some editable installation configurations.

munechika-koyo · 2025-07-18T16:43:47Z

Thank you for your suggestion @jacklovell
So, should we implement INFINITY as a cdef const double as well?

CnlPepper · 2025-07-18T16:45:45Z

Single floating point infinity promotes to double floating infinity, so there shouldn't be a need to cast it. Infinity has a special definition in IEEE float.

munechika-koyo · 2025-07-18T17:47:11Z

@CnlPepper Thank you for dedicating time to elevating Raysect to the next level.
Using the meson build system will boost build speed and enhance compilation efficiency, although it requires declarative setup of all modules.

If you implement such a feature soon, I will stop modifying the code in this PR and will close it as soon as another PR about Cython v3 compatibility appears.

…`INFINITY` in `libc.math` .

CnlPepper · 2025-07-19T23:02:07Z

You are too generous @munechika-koyo, I've been AWOL for quite a while!

I've just submitted an PR to move raysect to meson-python(see #446). I've build an automatic meson.build file generator to keep the management effort low. Would be great to get some feedback on it.

There is a lot of great work in your PR that I'd like to build from. I'm thinking of initially pulling this PR into a feature branch sitting on top of the meson branch. Once that goes in we can then get the move to cython 3 done. I should have some time to look at this tomorrow hopefully.

Final stage after we get cython 3 going is to rework the release system and get raysect v0.9.0 out to everyone.

CnlPepper · 2025-07-19T23:31:05Z

Work continues in: #447

vsnever reviewed Aug 1, 2023

View reviewed changes

vsnever approved these changes Aug 2, 2023

View reviewed changes

jacklovell requested changes Oct 28, 2024

View reviewed changes

Koyo Munechika and others added 7 commits November 27, 2024 11:20

add reverse method like __radd__, __rmul__, etc.

ea72675

This commit works in progress

fix inconsistent indentation

4095c99

fix binary arithmetic operators in math module

2d1a680

- mainly function base modules are modified

add noexcept to _edge_compare function

e9800f3

this modification is derived from the cython 3.0 compile error.

add ignoring numpy deplicated warning

245faf7

change to return from raise

b993ac6

update cython version constraint in pyproject.toml

7a12a04

munechika-koyo force-pushed the cython_v3_support branch from 2f7b3f8 to 7a12a04 Compare November 27, 2024 10:28

Mateasek mentioned this pull request Feb 4, 2025

Feature/function6d cherab/core#478

Merged

munechika-koyo added 4 commits July 18, 2025 13:56

fix: replace DEF statements with cdef enum, cdef const and use …

80a6f5f

…`INFINITY` in `libc.math` .

refactor: remove redundant DEF constants for R_2_PI and R_4_PI

4bf12d8

refactor: remove unused constants and clean up docstring formatting

a87eb2d

chore: update cython version constraint to 3.1 in pyproject.toml

1ad5da5

CnlPepper changed the base branch from development to feature/cython3 July 19, 2025 23:04

CnlPepper merged commit fb31ca6 into raysect:feature/cython3 Jul 19, 2025

CnlPepper mentioned this pull request Jul 19, 2025

Add Cython 3.1 support #447

Merged

munechika-koyo deleted the cython_v3_support branch July 22, 2025 18:13

	DEF NN = 312
	DEF MM = 156

	# The array for the state vector
	cdef uint64_t mt[NN]

Cython v3 support #426

Cython v3 support #426

Uh oh!

Conversation

munechika-koyo commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsnever left a comment

Choose a reason for hiding this comment

Uh oh!

munechika-koyo commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsnever left a comment

Choose a reason for hiding this comment

Uh oh!

jacklovell commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vsnever commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacklovell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jacklovell Oct 28, 2024

Choose a reason for hiding this comment

Uh oh!

munechika-koyo Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vsnever commented Oct 31, 2024

Uh oh!

munechika-koyo commented Jul 17, 2025

Uh oh!

jacklovell commented Jul 18, 2025

Uh oh!

munechika-koyo commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

munechika-koyo commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacklovell commented Jul 18, 2025

Uh oh!

CnlPepper commented Jul 18, 2025

Uh oh!

munechika-koyo commented Jul 18, 2025

Uh oh!

CnlPepper commented Jul 18, 2025

Uh oh!

munechika-koyo commented Jul 18, 2025

Uh oh!

CnlPepper commented Jul 19, 2025

Uh oh!

CnlPepper commented Jul 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

munechika-koyo commented Jul 28, 2023 •

edited

Loading

munechika-koyo commented Aug 2, 2023 •

edited

Loading

jacklovell commented Oct 20, 2023 •

edited

Loading

vsnever commented Oct 20, 2023 •

edited

Loading

munechika-koyo commented Jul 18, 2025 •

edited

Loading

munechika-koyo commented Jul 18, 2025 •

edited

Loading