__reduce__ no longer works on user defined types #17294
Description
I'm migrating from 1.13.3 to 1.17.4 and calling __reduce__ no longer works on our user defined types. Ultimately this stops me being able to pickle these objects. I've verified this problem is still present in the latest release 1.19.2 and the latest master.
Reproducing code example:
This can be demonstrated with the rational user defined type that is provided with numpy
python3
>>> import numpy.core._rational_tests
>>> numpy.core._rational_tests.rational(1,2).__reduce__()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: cannot include dtype 'r' in a buffer
On 1.13.3 this worked fine
python3
>>> import numpy.core.test_rational
>>> numpy.core.test_rational.rational(1,2).__reduce__()
(<built-in function scalar>, (dtype(rational), b'\x01\x00\x00\x00\x01\x00\x00\x00'))
Numpy/Python version information:
1.17.4 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0]
Analysis
Looks like this arises from migrating to the new buffer protocol in #10564 involving @vanossj and @eric-wieser This invokes a different code path that calls _buffer_format_string and this does not support user defined types. In fact, PyObject_GetBuffer is being called with PyBUF_SIMPLE so it won't even use the results from _buffer_format_string.
One possible fix is to support user defined types in buffer.c:_buffer_format_string
From
default:
PyErr_Format(PyExc_ValueError,
"cannot include dtype '%c' in a buffer",
descr->type);
return -1;
}
To
default:
if (PyTypeNum_ISUSERDEF(descr->type_num))
{
if (_append_char(str, descr->kind) < 0) return -1;
break;
}
PyErr_Format(PyExc_ValueError,
"cannot include dtype '%c' in a buffer",
descr->type);
return -1;
}