Variable sized Python objects become fully opaque in abi3t

[wjakob]

Bug report

Bug description:

The PyVarObject data structure becomes opaque in abi3t. Unfortunately, there is no limited API functionality to provide access to elements of a PyVarObject.

The relevant functions PyObject_GetItemData() and PyObject_GetItemData_DuringGC() are marked as unstable.

It would be very important to fix this before Python 3.15 goes out, or any extensions using variable sized objects will be stranded.

CPython versions tested on:

3.15, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-148325: Add PyObject_GetItemData() to the limited C API and stable ABI #148324

[encukou]

How do you define such objects, when PyVarObject is opaque?

[wjakob]

Using the PyType_*_FromSpec() API, with negative base size.

[encukou]

But, that assumes the base type is PyObject (unless you're deriving from PyVarObject already; then there's the same concern with that base type).
How are you handling the extra opaque field(s) needed for PyVarObject?

[ngoldbaum]

@wjakob - do you have a nanobind branch that implements abi3t support using the functions you’d like to stabilize, built against your branch of CPython? It might help clarify things of we can look at how you’d like to use the functions.

[wjakob]

You are right to call this out @encukou. I checked again and am doing something rather hacky (https://github.com/wjakob/nanobind/blob/abi3t/src/nb_internals.cpp#L248). I'm instantiating some dummy types and doing arithmetic to try to infer sizeof(PyObject) and sizeof(PyVarObject) to then set the PyType_Spec::basicsize with an absolute value. This happens to work, but the particular approach might not be correct.

Another potentially slightly less hacky idea would be to query the __basicsize__ of PyTuple_Type (which we know to be a PyVarObject). This could then similarly be used to set the PyType_Spec::basicsize of a new variable sized heap type (plus whatever is needed by the type on top). If the tuple's basic size ever becomes larger than a bare PyVarObject then this would lead to some waste.

    PyObject *attr = PyObject_GetAttrString(
        (PyObject *)&PyTuple_Type, "__basicsize__");
    if (attr == NULL) {
        return -1;
    }
    Py_ssize_t pvo_size = PyLong_AsSsize_t(attr);
    Py_DECREF(attr);
    if (pvo_size < 0) {
        return -1;
    }
    VarVec_spec.basicsize = (int)pvo_size;

    PyObject *type = PyType_FromSpec(&VarVec_spec);
....

If you have better ideas, then I am all ears.

Being able to allocate variable sized Python objects is important for nanobind. It implements a custom function object that can represent multiple C++ functions ("overloads"), and the variable size feature is used to dynamically expand to the actual number of present overloads.