Explore support for the opaque PyObject ABI

[ngoldbaum]

After #5753 is merged, it should be possible to experiment with the opaque PyObject ABI. I don't think we should merge this code into PyO3 (yet) but I think it's important to work through this exercise to enable end-to-end testing of PEP 803. My end goal is to build a abi3t wheel for cryptography by using the experimental private support in CPython for opaque PyObject via _Py_OPAQUE_PYOBJECT.

I'm planning to work on this but if anyone else wants to help out just let me know.

First, I'll need to expose _Py_OPAQUE_PYOBJECT in pyo3-build-config in a manner analogous to Py_GIL_DISABLED and Py_LIMITED_API.

After that, I'll need to update the FFI bindings to no longer assume that Py_LIMITED_API implies not(Py_GIL_DISABLED), which probably means updating some conditional compilation. I'll also need to update a few definitions for structs to match the CPython headers when _Py_OPAQUE_PYOBJECT is defined.

Finally, I'll need to update anywhere in PyO3 itself where we assume Py_LIMITED_API implies not(Py_GIL_DISABLED) or somehow relies on the layout of PyObject. Since PyO3 uses heap types and (after #5753) PEP 793 module initialization, I don't think it's necessary to update to use different C APIs or anything like that.

At that point, I should be able to actually build a limited API extension on the free-threaded build.

To get to the point where we can build limited API wheels, we might need to update code in maturin and setuptools-rust.

[ngoldbaum]

I hit a snag - there's a bunch of critical sections in PyO3 internals! This is good, since it means the internals are already written in a thread-safe manner but it's also bad because on the free-threaded build, we'd really want to have both Py_GIL_DISABLED and Py_LIMITED_API exposed, but the PyCriticalSection C structs aren't available in the limited API. I think I might write variants that are no-ops just so I can make progress but it's not going to be thread-safe without a critical section that acts like the critical section on the free-threaded build. And that needs support from CPython, I think.

[ngoldbaum]

I wrote about this in the capi workgroup issue: capi-workgroup/decisions#92 (comment)

[ngoldbaum]

#5807 gets opaque PyObject almost fully working on the GIL-enabled build. There are still issues around subclassing that need to be fixed.