_DuringGC functions

[encukou]

There are newly documented restrictions on tp_traverse:

The traversal function must not have any side effects. It must not modify the reference counts of any Python objects nor create or destroy any Python objects.

But, whether a C-API function modifies some reference count is an implementation detail that we can change at any time. Core devs can add logging, an audit event, or a warning, to virtually any function.
This implies that you can't safely use any C API function in tp_traverse, unless it guarantees that it has no side effects.

I propose to add such side-effect-free versions of some functions useful in tp_traverse, with a _DuringGC suffix -- namely:

• PyObject_GetTypeData_DuringGC
• PyObject_GetItemData_DuringGC
• PyType_GetModuleState_DuringGC
• PyModule_GetState_DuringGC
• PyModule_GetToken_DuringGC
• PyType_GetBaseByToken_DuringGC
• PyType_GetModule_DuringGC
• PyType_GetModuleByToken_DuringGC

Compared to their non-suffixed variants, the main difference is that we promise that these will remain safe to use in tp_traverse. The other differences are that these functions cannot fail, and (where applicable) they return borrowed references rather than strong ones.

Also, Py_VISIT, Py_SIZE, PyObject_VisitManagedDict as well as the visitproc that Python passes to tp_traverse will be documented to work like this.

Users that call any _DuringGC function (including some type's tp_traverse) must manually must ensure that the type, MRO, and layout of any argument doesn't concurrently change. That means only calling them from tp_traverse.

---

See the discussion (and preferably join that if you have input): https://discuss.python.org/t/adding-c-api-for-use-in-tp-traverse/105331/13

(Note that I agree that preferring all functions to be fallible makes things less ergonomic. But being consistent is important, and ergonomics issues are exactly when we add suffixed variants of functions -- _DuringGC in this case.)

[vstinner]

The other differences are that these functions cannot fail

What is PyType_GetModuleState_GCSafe() behavior if the type has no associated module state for example? Does it return NULL? What do you mean by "cannot fail"?

[encukou]

Yeah, that should be clarified; thanks for the question.

Let's do this: They all do not set the exception, but otherwise have the same API. So, PyType_GetModuleState_GCSafe and most others return NULL without setting an exception.
The ones that return int (GetToken, GetBaseByToken) set *result to NULL and return -1. You can't do much with the -1, but I don't think the _GCSafe variant should use a different signature.

Does that sound good?

[vstinner]

They all do not set the exception, but otherwise have the same API.
Does that sound good?

Thanks, that's what I expected.

[encukou]

Alright, let's see where we stand.

• @encukou
• @erlend-aasland
• @serhiy-storchaka
• @vstinner
• @zooba

[encukou]

ping @erlend-aasland, @serhiy-storchaka

Note that there's a discussion on the naming in the discussion thread.

[erlend-aasland]

I prefer to let the discussion come to a conclusion before voting, thanks.

[encukou]

That's fair.
But note that the discussion is about naming only. If you have any other concerns, it would be great if you can bring them up now.

[erlend-aasland]

Sure, but naming is important.

[erlend-aasland]

It seems like the majority agrees on _DuringGC. I won't object to that.

[encukou]

Updated to _DuringGC. (The people who already voted agreed with the name.)

ping @serhiy-storchaka @zooba

[encukou]

ping @serhiy-storchaka