Compilation failure with Python 3.11

[SnarkBoojum]

I wanted to update the Debian package for fpylll to the latest version because we want to transition to Python 3.12 ; unfortunately during the transition that means we compile for 3.11 and 3.12.

And with 3.11 I get a failure:

In function ‘PyObject* __Pyx_PyInt_From_int(int)’,
    inlined from ‘PyObject* __pyx_pf_6fpylll_5fplll_3bkz_12BKZReduction_6__call__(__pyx_obj_6fpylll_5fplll_3bkz_BKZReduction*)’ at build/src/fpylll/fplll/bkz.cpp:4422:35:
build/src/fpylll/fplll/bkz.cpp:15227:34: warning: ‘__pyx_v_r’ may be used uninitialized [-Wmaybe-uninitialized]
15227 |             return PyInt_FromLong((long) value);
      |                                  ^
build/src/fpylll/fplll/bkz.cpp: In function ‘PyObject* __pyx_pf_6fpylll_5fplll_3bkz_12BKZReduction_6__call__(__pyx_obj_6fpylll_5fplll_3bkz_BKZReduction*)’:
build/src/fpylll/fplll/bkz.cpp:4095:7: note: ‘__pyx_v_r’ was declared here
 4095 |   int __pyx_v_r;
      |       ^~~~~~~~~
build/src/fpylll/fplll/enumeration.cpp:2101:21: error: conflicting declaration of ‘bool evaluator_callback_call_obj(PyObject*, int, double*)’ with ‘C’ linkage
 2101 | __PYX_EXTERN_C bool evaluator_callback_call_obj(PyObject *, int, double *); /*proto*/
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from build/src/fpylll/fplll/enumeration.cpp:874:
build/src/fpylll/fplll/enumeration_callback_helper.h:10:8: note: previous declaration with ‘C++’ linkage
   10 |   bool evaluator_callback_call_obj(PyObject *obj, int n, double *new_sol_coord);
      |        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/lib/ccache/x86_64-linux-gnu-gcc' failed with exit code 1
E: pybuild pybuild:395: build: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py build 
dh_auto_build: error: pybuild --build -i python{version} -p 3.11 returned exit code 13

My first idea is "why two declarations?", but I haven't dug to see if that was indeed the problem.

[malb]

There seems to be a conflict between Cython < 3 and Cython 3 on how this function should be declared:

https://github.com/fplll/fpylll/blob/21123b5a88a5d0672f172b824925b00e3839b7c0/src/fpylll/fplll/enumeration_callback_helper.h#L8C1-L11C5

I have no good idea how to handle this? Anyone got any ideas?

[SnarkBoojum]

Well, Debian wants to package a newer version of cython -- that's why I'm looking into this. That might get around the issue...

[joerowell]

@malb The issue is that the cdef public... function declaration introduces a definition (and hence also a declaration) into the program from the perspective of the C++ compiler (this is basically why we get the "duplicate declaration" error in the logs above: it's already declared). Looking at the error logs, it looks like this used to be a C declaration, but now (for some reason) it's a C++ declaration, and so they don't always line up correctly. For example, if I extract everything out into its own file, I get:

// callback_func.h
#ifdef CYTHON_EXTERN_C
    #undef __PYX_EXTERN_C
    #define __PYX_EXTERN_C CYTHON_EXTERN_C
#elif defined(__PYX_EXTERN_C)
    #ifdef _MSC_VER
    #pragma message ("Please do not define the '__PYX_EXTERN_C' macro externally. Use 'CYTHON_EXTERN_C' instead.")
    #else
    #warning Please do not define the '__PYX_EXTERN_C' macro externally. Use 'CYTHON_EXTERN_C' instead.
    #endif
#else
  #ifdef __cplusplus
    #define __PYX_EXTERN_C extern "C"
  #else
    #define __PYX_EXTERN_C extern
  #endif
#endif

__PYX_EXTERN_C bool evaluator_callback_call_obj(PyObject *, int, double *);

I think the best thing to do is just to extract the callback function into its own .pyx file:

# callback_func.pyx
from libcpp cimport bool

cdef public bool evaluator_callback_call_obj(obj, int n, double *new_sol_coord):
    cdef list new_sol_coord_ = []
    for i in range(n):
        new_sol_coord_.append(new_sol_coord[i])
    return obj(new_sol_coord_);

And then just include the resulting header in callback_helper.h:

#include "callback_func.h"

This is basically what's described here.

That way, we don't need to worry about the linkage type, because it should always be correct. You might need to play around with the include paths, though, since callback_func.h is auto-generated.