BUG: PyDataType_*, PyArray_* functions undeclared with meson / Numpy 2.0 build

[matthew-brett]

Describe the issue:

Over at https://github.com/nipy/nipy - out Meson-based build works without error on Numpy 1.26.4. But, in the same virtual environment, with the single change of installing Numpy 2.0 - we get multiple build errors of form:

nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3042:13: error: call to undeclared function 'PyDataType_ELSIZE'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyDataType_ELSIZE(__pyx_v_self);

with many Numpy-defines missing - PyDataType_SUBARRAY, PyDataType_FIELDS, PyArray_MultiIter_NUMITER and so on. These are all in a generated Cython file, using Cython 3.0.10.

Is it possible I missed some upgrade to Cython or similar that includes such defines for Numpy 2.0?

Reproduce the code example:

git clone https://github.com/nipy/nipy
cd nipy
pip install cython

followed by:

pip install .

or:

meson build
cd build
ninja -j 1

Error message:

[7/68] Compiling C object nipy/algorithms/graph/_g...cpython-311-darwin.so.p/meson-generated__graph.c.o
FAILED: nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/meson-generated__graph.c.o 
cc -Inipy/algorithms/graph/_graph.cpython-311-darwin.so.p -Inipy/algorithms/graph -I../nipy/algorithms/graph -I/Users/mb312/Library/Python/3.11/lib/python/site-packages/numpy/core/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -fvisibility=hidden -fcolor-diagnostics -Wall -Winvalid-pch -std=c99 -O2 -g -Wno-unused-function -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -MD -MQ nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/meson-generated__graph.c.o -MF nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/meson-generated__graph.c.o.d -o nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/meson-generated__graph.c.o -c nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3042:13: error: call to undeclared function 'PyDataType_ELSIZE'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyDataType_ELSIZE(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3076:13: error: call to undeclared function 'PyDataType_ALIGNMENT'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyDataType_ALIGNMENT(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3114:15: error: call to undeclared function 'PyDataType_FIELDS'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_t_1 = PyDataType_FIELDS(__pyx_v_self);
              ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3114:13: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
  __pyx_t_1 = PyDataType_FIELDS(__pyx_v_self);
            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3156:15: error: call to undeclared function 'PyDataType_NAMES'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_t_1 = PyDataType_NAMES(__pyx_v_self);
              ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3156:13: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
  __pyx_t_1 = PyDataType_NAMES(__pyx_v_self);
            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3194:13: error: call to undeclared function 'PyDataType_SUBARRAY'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyDataType_SUBARRAY(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3194:11: warning: incompatible integer to pointer conversion assigning to 'PyArray_ArrayDescr *' (aka 'struct _arr_descr *') from 'int' [-Wint-conversion]
  __pyx_r = PyDataType_SUBARRAY(__pyx_v_self);
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3228:13: error: call to undeclared function 'PyDataType_FLAGS'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyDataType_FLAGS(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3262:13: error: call to undeclared function 'PyArray_MultiIter_NUMITER'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_NUMITER(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3296:13: error: call to undeclared function 'PyArray_MultiIter_SIZE'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_SIZE(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3330:13: error: call to undeclared function 'PyArray_MultiIter_INDEX'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_INDEX(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3364:13: error: call to undeclared function 'PyArray_MultiIter_NDIM'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_NDIM(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3398:13: error: call to undeclared function 'PyArray_MultiIter_DIMS'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_DIMS(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3398:11: warning: incompatible integer to pointer conversion assigning to 'npy_intp *' (aka 'long *') from 'int' [-Wint-conversion]
  __pyx_r = PyArray_MultiIter_DIMS(__pyx_v_self);
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3432:13: error: call to undeclared function 'PyArray_MultiIter_ITERS'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  __pyx_r = PyArray_MultiIter_ITERS(__pyx_v_self);
            ^
nipy/algorithms/graph/_graph.cpython-311-darwin.so.p/_graph.c:3432:11: warning: incompatible integer to pointer conversion assigning to 'void **' from 'int' [-Wint-conversion]
  __pyx_r = PyArray_MultiIter_ITERS(__pyx_v_self);
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Python and NumPy Versions:

2.0.0
3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Runtime Environment:

[{'numpy_version': '2.0.0',
  'python': '3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 15.0.0 '
            '(clang-1500.1.0.2.5)]',
  'uname': uname_result(system='Darwin', node='nipraxis.dynevor.org', release='23.5.0', version='Darwin Kernel Version 23.5.0: Wed May  1 20:19:05 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8112', machine='arm64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP'],
                      'not_found': ['ASIMDFHM']}}]

Context for the issue:

Blocking port to Numpy 2.0.

[seberg]

Strange, that would look like you are using old headers but new the pxd files. I am wondering if this is on a debian based system or so, I used to have the problem that debian shipped the NumPy headers a bit too generally, I don't have it right now... but that might be because I deleted them long ago.
(I do believe this was fixd in debian a while ago, at last I seem to remember an email that the issue was closed)

EDIT: Sorry, missed the "Darwin" part, but the problem still seems to be that wrong headers get picked up

[seberg]

When I run it locally, I do get errors due to PyArray_CastTo not existing anymore: needs to be PyArray_CopyInto now (there where three identical functions).

But fixing that, things seem to build fine. (not sure how well the pip install works, presumably it does with --no-build-isolation --no-deps is probably fine.)

[matthew-brett]

Aha - yes - it seems that the more profuse warnings are due to some missing installs in the virtualenv - I have not tracked down which - but I can now build with the CastTo changes via pip install . or spin build - thanks.