'nogil' keyword / GIL issues on RHEL8 using Python 3.11

[ralphlange]

python3.11 -m cython -I ../p4p -o _p4p.cpp --cplus ../p4p/_p4p.pyx
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/pvxs/data.pxd:96:15: Rvalue-reference as function argument not supported
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/pvxs/data.pxd:111:13: Rvalue-reference as function argument not supported
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/pvxs/data.pxd:96:15: Rvalue-reference as function argument not supported
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/pvxs/data.pxd:111:13: Rvalue-reference as function argument not supported
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/_p4p.pyx:45:72: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.
[ERROR] performance hint: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/_gw.pyx:351:4: Exception check on 'GWProvider_makeChannel' will always require the GIL to be acquired. Declare 'GWProvider_makeChannel' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
[WARN] performance hint: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/_gw.pyx:374:4: Exception check on 'GWProvider_audit' will always require the GIL to be acquired.
[WARN] Possible solutions:
[ERROR] 1. Declare 'GWProvider_audit' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
[ERROR] 2. Use an 'int' return type on 'GWProvider_audit' to allow an error code to be returned.
[WARN] warning: /mnt/ITER/langer/work/CODAC/m-epics-p4p/target/p4p-4.2.1/src/p4p/pvxs/sharedArray.pxd:28:20: Rvalue-reference as function argument not supported

[mdavidsaver]

It looks like these errors/warnings begin with cython 3.

With Debian 13 (cpy 3.13.5, cython 3.0.11) I see:

warning: /home/dev/epics/p4p/src/p4p/pvxs/data.pxd:96:15: Rvalue-reference as function argument not supported
warning: /home/dev/epics/p4p/src/p4p/pvxs/data.pxd:111:13: Rvalue-reference as function argument not supported
warning: /home/dev/epics/p4p/src/p4p/_p4p.pyx:47:72: The keyword 'nogil' should appear at the end of the function signature line. Placing it before 'except' or 'noexcept' will be disallowed in a future version of Cython.
warning: /home/dev/epics/p4p/src/p4p/pvxs/sharedArray.pxd:28:20: Rvalue-reference as function argument not sup
...

The "nogil" ordering is a trivial change. The definitions using rval reference are (I think) unused from cython code.

...
performance hint: /home/dev/epics/p4p/src/p4p/_gw.pyx:351:4: Exception check on 'GWProvider_makeChannel' will always require the GIL to be acquired. Declare 'GWProvider_makeChannel' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
performance hint: /home/dev/epics/p4p/src/p4p/_gw.pyx:374:4: Exception check on 'GWProvider_audit' will always require the GIL to be acquired.
Possible solutions:
        1. Declare 'GWProvider_audit' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
        2. Use an 'int' return type on 'GWProvider_audit' to allow an error code to be returned.

I guess these "performance hint" got promoted to "don't do this" at some point after cython 3.0.11.

Adding a noexcept qualifier is easy for GWProvider_audit(), which already swallows exceptions.

I think the same is true of GWProvider_makeChannel(). From looking at the generated C++ code, it looks like adding noexcept has no effect on the output, with the exception behavior being to return a default constructed shared_ptr (aka. nullptr), which the caller is checking for.

backwards compatibility of the noexcept modifier with older cython may also be an issue.

[mdavidsaver]

backwards compatibility of the noexcept modifier with older cython may also be an issue.

Yup...

     Error compiling Cython file:
    ------------------------------------------------------------
    ...
            try:
                return handler.testChannel(channel, peer.decode())
            except:
                return GWSearchBanHost

        shared_ptr[GWChan] GWProvider_makeChannel(GWSource* src, unique_ptr[ChannelControl]* op) noexcept with gil:
                                                                                                ^
    ------------------------------------------------------------

    src/p4p/_gw.pyx:351:93: Syntax error in C variable declaration

[mdavidsaver]

As I read it, the noexcept keyword was added with cython 3.0 and backported as a no-op in 0.29.31. So excepting some code generation games, adding noexcept to fix this issue means requiring Cython >= 0.29.31. or omit noexcept and require Cython < 3.1.0 (not a long term solution).

The current stated minimum is Cython>=0.20.

I suspect that Cython >= 0.29.31 would be ok for wheel builds. Also for Debian from 12 (oldstable). I have not investigated the RHEL situation.

@coretl @ralphlange @AlexanderWells-diamond thoughts?

[AlexanderWells-diamond]

On RHEL8 with Python3.8, Cython 3.2.3 I see these build warnings:

warning: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/pvxs/data.pxd:96:15: Rvalue-reference as function argument not supported
warning: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/pvxs/data.pxd:111:13: Rvalue-reference as function argument not supported
performance hint: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/_gw.pyx:351:4: Exception check on 'GWProvider_makeChannel' will always require the GIL to be acquired. Declare 'GWProvider_makeChannel' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
performance hint: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/_gw.pyx:374:4: Exception check on 'GWProvider_audit' will always require the GIL to be acquired.

And with Python2.7, Cython 3.0.12 I see a very similar set:

warning: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/pvxs/data.pxd:96:15: Rvalue-reference as function argument not supported
warning: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/pvxs/data.pxd:111:13: Rvalue-reference as function argument not supported
performance hint: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/_gw.pyx:351:4: Exception check on 'GWProvider_makeChannel' will always require the GIL to be acquired. Declare 'GWProvider_makeChannel' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
performance hint: /scratch/eyh46967/dev/epics_python/p4p/src/p4p/_gw.pyx:374:4: Exception check on 'GWProvider_audit' will always require the GIL to be acquired.
Possible solutions:
        1. Declare 'GWProvider_audit' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
        2. Use an 'int' return type on 'GWProvider_audit' to allow an error code to be returned.

And with Python2.7, Cython 0.29.37 I see no warnings at all.

We would want to specify Cython >= 0.29.32, as 0.29.31 was yanked from PyPi. I don't see an issue with this; I'm not terribly au fait with Cython intricacies, but the entire 0.29.* release series still supports Python2. In fact everything up to 3.0.12 supports Python2.

Note that the current documentation specifies a Cython version of >=0.25.2, and all of the requirements-*.txt files specify an even higher version of Cython. So there's some ambiguity about what is the definite minimum version.

[mdavidsaver]

Should be resolved by 14f2b09. f612608 updates the Cython dependency.