[WIP] Support passing package name as pythran signature #2371

serge-sans-paille · 2025-11-15T23:10:41Z

This patch introduces a new argument type of the form <name> pkg, for instance numpy pkg.
It's a special function export type which, if used, must be used
consistently across all overloads for that exported function.

It is resolved before any compilation step and is syntactically
equivalent to statically resolving the given argument to the given
package. For instance,

#pythran export foo(float, numpy pkg)
def foo(x, np):
    return np.cos(x)

is equivalent to the following:

#pythran export foo(float, numpy pkg)
def foo(x, _):
    import numpy as np
    return np.cos(x)

the exported foo function still has two parameters and the second
parameter is going to check that the passed argument is a module named
'numpy'.

Fix #2367

serge-sans-paille · 2025-11-15T23:11:40Z

@ev-br : This is a work-in progress, can you have a look at the commit message and confirm it matches your needs?

codspeed-hq · 2025-11-16T00:04:56Z

CodSpeed Performance Report

Merging #2371 will not alter performance

_{Comparing bug/2367 (060e174) with master (b00edbe)}

Summary

✅ 58 untouched

ev-br · 2025-11-16T13:22:29Z

Yes, it looks just like what a doctor's ordered!

The np = import numpy example gave me a pause at first, but indeed the intent is clear and as far as commit messages or short documentation goes, it's great actually.

serge-sans-paille · 2025-11-16T13:23:31Z

Yes, it looks just like what a doctor's ordered!

The np = import numpy example gave me a pause at first, but indeed the intent is clear and as far as commit messages or short documentation goes, it's great actually.

^^! my bad I meant import numpy as np of course

serge-sans-paille · 2025-11-16T13:26:04Z

@ev-br btw, pythran will still fail in the following case:

#pythran export foo(x, numpy pkg)
def foo(x, np):
    bar(x, np)

def bar(x, np):
    return np.cos(x)  # pythran currently does not support passing package to helper functions

Would that be an issue in you scenario? I guess that's something I can fix, but it won't be trivial

ev-br · 2025-11-16T14:20:22Z

pythran currently does not support passing package to helper functions
Would that be an issue in you scenario?

I'm not entirely sure how pythran deals with helper functions (what must be annotated and what is inferred automagically), but AFAIU, yes, something like this will be a blocker.

The target I'm after is to pythranize https://github.com/scipy/scipy/blob/main/scipy/interpolate/_rbfinterp_xp.py#L213C1-L215C3
(compare to curretnly pythranized https://github.com/scipy/scipy/blob/main/scipy/interpolate/_rbfinterp_pythran.py)

The meat of the matter is this:

# pythran export _build_evaluation_coefficients(float[:, :],
#                          float[:, :],
#                          str,
#                          float,
#                          int[:, :],
#                          float[:],
#                          float[:]
#                          numpy pkg)
def _build_evaluation_coefficients(
    x, y, kernel, epsilon, powers, shift, scale, xp
):
    kernel_func = NAME_TO_FUNC[kernel]
    # ..... snip

    kernel_func(
        xp.linalg.vector_norm(x[:, None, :] - y[None, :, :], axis=-1),
        xp
    ),

where kernel_func needs the xp argument (https://github.com/scipy/scipy/blob/main/scipy/interpolate/_rbfinterp_xp.py#L127):

NAME_TO_FUNC = {
   "multiquadratic": lambda r, xp: -xp.sqrt(r**2 + 1),
  "linear": lambda r, x: -r,
  # ... snip
}

Also, does this part of the commit message mean there's a runtime check of the __name__ attribute?

and the second parameter is going to check that the passed argument is a module named 'numpy'.

serge-sans-paille · 2025-11-16T14:25:37Z

pythran currently does not support passing package to helper functions
Would that be an issue in you scenario?

I'm not entirely sure how pythran deals with helper functions (what must be annotated and what is inferred automagically), but AFAIU, yes, something like this will be a blocker.

Nothing needs to be annotated for non exported functions. I'll think of a way to deal with that.

Also, does this part of the commit message mean there's a runtime check of the __name__ attribute?

Yes!

Using serge-sans-paille/pythran#2371

serge-sans-paille · 2025-12-07T14:33:52Z

@ev-br : updated approach, still work in progress, but should be fairly stable now. It works on your reproducer with a slight s/concat/concatenate/

ev-br · 2025-12-08T10:53:39Z

Great!

Tested it locally --- just two glitches:

Compiling the "full" example from Allow annotating a function argument as numpy #2367 (comment) fails with AttributeError: 'Type' object has no attribute 'generate'. Did you mean: 'sgenerate'?. I first thought it's related to passing a callable in kernel_matrix, but apparently not. The full error is:

$ pythran scipy/interpolate/_rbfinterp_pythran.py
Traceback (most recent call last):
  File "/home/br/miniforge3/envs/scipy-dev/bin/pythran", line 7, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/br/repos/pythran/pythran/run.py", line 190, in run
    pythran.compile_pythranfile(args.input_file,
  File "/home/br/repos/pythran/pythran/toolchain.py", line 550, in compile_pythranfile
    output_file = compile_pythrancode(module_name, fd.read(),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/toolchain.py", line 451, in compile_pythrancode
    module, error_checker = generate_cxx(module_name, pythrancode, specs, opts,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/toolchain.py", line 147, in generate_cxx
    content = pm.dump(Cxx, ir)
              ^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/passmanager.py", line 229, in dump
    return b.run(node)
           ^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/passmanager.py", line 148, in run
    return super(ModuleAnalysis, self).run(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/passmanager.py", line 128, in run
    super(Analysis, self).run(node)
  File "/home/br/repos/pythran/pythran/passmanager.py", line 107, in run
    return self.visit(node)
           ^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/passmanager.py", line 89, in visit
    return super(ContextManager, self).visit(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/miniforge3/envs/scipy-dev/lib/python3.12/ast.py", line 407, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 1421, in visit_Module
    decls_n_defns = list(filter(None, (self.visit(stmt) for stmt in
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 1421, in <genexpr>
    decls_n_defns = list(filter(None, (self.visit(stmt) for stmt in
                                       ^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/passmanager.py", line 89, in visit
    return super(ContextManager, self).visit(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/miniforge3/envs/scipy-dev/lib/python3.12/ast.py", line 407, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 1433, in visit_FunctionDef
    return visitor.visit(node)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 252, in visit
    result = super(CxxFunction, self).visit(node)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/br/miniforge3/envs/scipy-dev/lib/python3.12/ast.py", line 407, in visit
    return visitor(node)
           ^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 370, in visit_FunctionDef
    ctx(result_type),
    ^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 129, in __call__
    t = node.generate(self)
        ^^^^^^^^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/cxxtypes.py", line 323, in generate
    return f'typename pythonic::returnable<{ctx(self.of)}>::type'
                                            ^^^^^^^^^^^^
  File "/home/br/repos/pythran/pythran/backend.py", line 129, in __call__
    t = node.generate(self)
        ^^^^^^^^^^^^^
AttributeError: 'Type' object has no attribute 'generate'. Did you mean: 'sgenerate'?

slight s/concat/concatenate/

Ah, this numpy 2.0 / array api spelling would actually be nice to support (I know I keep and keep asking things...). Split it off to #2379 for convenience.

serge-sans-paille · 2025-12-09T17:05:31Z

Patch rebased on #2379 and updated, it now compiles your original source !

serge-sans-paille · 2025-12-12T15:09:53Z

@ev-br can you confirm the last iteration is functional?

Using serge-sans-paille/pythran#2371

ev-br · 2025-12-14T11:08:11Z

Great! Testing it now..... It builds!

I'm now testing it in https://github.com/scipy/scipy/compare/main...ev-br:rbf_xp_dedupe?expand=1
The plan is to make it pass the scipy test suite, then benchmark (it previously used loops, the "dedupe" version needs to use vectorized calls, so it might put some more pressure on pythran compilation/optimization passes; I'll put it through its paces and report here, but that's for a bit later).

A first hiccup:

>       lhs, rhs, shift, scale = _build_system(
            y, d, smoothing, kernel, epsilon, powers, xp
            )
E       TypeError: Invalid call to pythranized function `_build_system(float64[:, :], float64[:, :] (is a view), float64[:], str, float, int64[:, :], module)'
E       Candidates are:
E       
E           - _build_system(float[:,:], float[:,:], float[:], str, float, int64[:,:], numpy pkg)

.... snip ...

scipy/interpolate/_rbfinterp_np.py:56: TypeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>
> /home/br/repos/scipy/scipy/build-install/usr/lib/python3.12/site-packages/scipy/interpolate/_rbfinterp_np.py(56)_build_and_solve_system()
-> lhs, rhs, shift, scale = _build_system(
(Pdb) l
 51  	        Domain shift used to create the polynomial matrix.
 52  	    scale : (N,) float ndarray
 53  	        Domain scaling used to create the polynomial matrix.
 54  	
 55  	    """
 56  ->	    lhs, rhs, shift, scale = _build_system(
 57  	        y, d, smoothing, kernel, epsilon, powers, xp
 58  	        )
 59  	    _, _, coeffs, info = dgesv(lhs, rhs, overwrite_a=True, overwrite_b=True)
 60  	    if info < 0:
 61  	        raise ValueError(f"The {-info}-th argument had an illegal value.")
(Pdb) p xp
<module 'scipy._lib.array_api_compat.numpy' from '/home/br/repos/scipy/scipy/build-install/usr/lib/python3.12/site-packages/scipy/_lib/array_api_compat/numpy/__init__.py'>
(Pdb) p xp.__name__
'scipy._lib.array_api_compat.numpy'

What happens here: array_api_compat.numpy is a smal wrapper over numpy, for the pythran purposes it's just numpy. That it's vendored into the scipy source is an implementation detail, but it can be vendored or installed standalone.
Would it be possible to relax the __name__ check? Either allow "numpy" and "array_api_compat.numpy", or .endswith("numpy") --- no functional changes, it's just numpy either way.

serge-sans-paille · 2025-12-14T14:55:56Z

Can you give it a try with the latest version of the branch and using array_api_compat.numpy pkg as numpy in the signatures instead of numpy pkg? It should tell pythran that the array_api_compat.numpypackage is expected, but it behaves as numpy`

ev-br · 2025-12-14T16:49:54Z

Erm... Something's amiss with the last version:

$ pythran -E scipy/interpolate/_rbfinterp_pythran.py
CRITICAL: I am in trouble. Your input file does not seem to match Pythran's constraints...
scipy/interpolate/_rbfinterp_pythran.py:9:38 error: Unexpected token `.` at that point.
----
    # NB: changed w.r.t. pythran, vectorized
                                      ^~~~ (o_0)
----

If I delete the offending line (which is a comment but OK), I get the same error on the next line.

The file is from this branch: https://github.com/scipy/scipy/compare/main...ev-br:rbf_xp_dedupe?expand=1

ev-br · 2025-12-14T16:52:41Z

Also, should it be array_api_compat.numpy pkg as numpy or scipy._lib.array_api_compat.numpy pkg as numpy? Based on

(Pdb) p xp.name
'scipy._lib.array_api_compat.numpy'

The latter definitely starts looking a bit strange for hardcoding a filesystem path into the source.

EDIT: all in all, I'd question the value of the __name__ check. Sure as a user I appreciate the system helping me dodge a footgun; that said, I'm fine if the docs just say "the annotated module is treated as being numpy by the compiler; using anything else at runtime is UB" I'd accept responsibility and probably throw a python-level assertion at the call site.
Which is much nicer with array_api_compat, actually: if not is_numpy(xp): raise ValueError.

serge-sans-paille · 2025-12-14T18:19:14Z

Patch updated, there were quite a few flaws. Thanks for the quick feedback!
Indeed currently you would need to use scipy._lib.array_api_compat.numpy. I've discarded this as clause and only requires that the trailing pathes matches, that seems a decent workaround.
So basically passing numpy or package whose names ends with .numpy should work.

ev-br · 2025-12-14T18:21:30Z

Thank you, will test-drive it! ETA likely a couple of days though (such as is the time of the year)

ev-br · 2025-12-15T12:06:04Z

With d8dee70, I get a bizzare

$ spin build
$ meson compile -j 4 -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /home/br/miniforge3/envs/scipy-dev/bin/ninja -C /home/br/repos/scipy/scipy/build -j 4
ninja: Entering directory `/home/br/repos/scipy/scipy/build'
[51/52] Compiling C++ object scipy/interpo....p/meson-generated__rbfinterp_pythran.cpp.o
FAILED: [code=1] scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated__rbfinterp_pythran.cpp.o 
/home/br/miniforge3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ -Iscipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p -Iscipy/interpolate -I../scipy/interpolate -I../../../../miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran -I../../../../miniforge3/envs/scipy-dev/lib/python3.12/site-packages/numpy/_core/include -I/home/br/miniforge3/envs/scipy-dev/include/python3.12 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -D_GLIBCXX_ASSERTIONS=1 -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++17 -O2 -g -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_NONE -Wno-cpp -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-unused-function -Wno-unused-variable -Wno-int-in-bool-context -MD -MQ scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated__rbfinterp_pythran.cpp.o -MF scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated__rbfinterp_pythran.cpp.o.d -o scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/meson-generated__rbfinterp_pythran.cpp.o -c scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/_rbfinterp_pythran.cpp
In file included from scipy/interpolate/_rbfinterp_pythran.cpython-312-x86_64-linux-gnu.so.p/_rbfinterp_pythran.cpp:18:
../../../../miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/pkg/numpy.hpp: In static member function 'static bool {anonymous}::pythonic::from_python<{anonymous}::pythonic::types::pkg::numpy>::is_convertible(PyObject*)':
../../../../miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/pkg/numpy.hpp:14:14: error: duplicate 'const'
   14 |   const char const *pkg_name = PyModule_GetName(obj);
      |              ^~~~~
      |              -----
ninja: build stopped: subcommand failed.

Also, I think I can now work around array_api_compat.numpy vs pristine numpy, so if it continues to be a brittle pain in the neck, we can roll back to it being just numpy (sorry for asking to a wild goose chase).

serge-sans-paille · 2025-12-15T12:10:42Z

On Mon, Dec 15, 2025 at 04:06:25AM -0800, Evgeni Burovski wrote: Also, I think I can now work around array_api_compat.numpy vs pristine numpy, so if it continues to be a brittle pain in the neck, we can roll back to it being just numpy (sorry for asking to a wild goose chase).

Should be good now (?)

ev-br · 2025-12-15T12:16:54Z

Should be good now (?)

Confirmed, thanks! With 278a39e scipy builds again (and fails all around, but that's on me for now at least). I'll keep experimenting on the user side.

serge-sans-paille · 2025-12-15T12:18:06Z

Great! I leave that PR open for a while, just in case we need to adjust based on your feedback.

ev-br · 2025-12-15T15:49:17Z

I seem to be missing something simple: I should be able to compile np.concatenate ( == np.concat), right?

(scipy-dev) br@gonzales:~/temp$ cat pythr.py 
import numpy as np

# pythran export try_1(float[:, :], float[:, :])
def try_1(x, y):
    z2 = np.concatenate((x, y), axis=-1)
    return z2
(scipy-dev) br@gonzales:~/temp$ pythran pythr.py
In file included from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/ndarray.hpp:39,
                 from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/tuple.hpp:8,
                 from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/builtins/bool_.hpp:6,
                 from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/NoneType.hpp:6,
                 from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/slice.hpp:5,
                 from /home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/core.hpp:52,
                 from /tmp/tmp6fogbgsc.cpp:1:
/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/numpy_gexpr.hpp: In instantiation of '{anonymous}::pythonic::types::numpy_gexpr<A, S>::numpy_gexpr(const {anonymous}::pythonic::types::numpy_gexpr<Argp, S ...>&) [with Argp = const {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> >&; Arg = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> >&; S = {{anonymous}::pythonic::types::cstride_normalized_slice<1>, long int}]':
/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/numpy/concatenate.hpp:67:28:   required from 'void {anonymous}::pythonic::numpy::details::concatenate_helper<N>::operator()(Out&&, const A&, long int, std::index_sequence<Is ...>) const [with Out = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&; A = {anonymous}::pythonic::types::array_base<{anonymous}::pythonic::types::numpy_texpr<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> > >, 2, {anonymous}::pythonic::types::tuple_version>; long unsigned int ...I = {0, 1}; long unsigned int N = 2; std::index_sequence<Is ...> = std::integer_sequence<long unsigned int, 0, 1>]'
   67 |                 difroms = {*std::get<I>(ifroms)...};
      |                            ^~~~~~~~~~~~~~~~~~~~
/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/numpy/concatenate.hpp:149:37:   required from '{anonymous}::pythonic::types::ndarray<typename E::dtype, {anonymous}::pythonic::types::array_base<long int, E::value, {anonymous}::pythonic::types::tuple_version> > {anonymous}::pythonic::numpy::concatenate(const {anonymous}::pythonic::types::array_base<T, N, V>&, long int) [with E = {anonymous}::pythonic::types::numpy_texpr<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> > >; long unsigned int M = 2; V = {anonymous}::pythonic::types::tuple_version; typename E::dtype = double]'
  149 |     details::concatenate_helper<N>()(out, args, axis, std::make_index_sequence<M>{});
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/include/numpy/concatenate.hpp:24:3:   required from 'decltype ({anonymous}::pythonic::numpy::concatenate((forward<Types>)(<unnamed>::pythonic::numpy::functor::concatenate::operator()::types)...)) {anonymous}::pythonic::numpy::functor::concatenate::operator()(Types&& ...) const [with Types = {{anonymous}::pythonic::types::array_base<{anonymous}::pythonic::types::numpy_texpr<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> > >, 2, {anonymous}::pythonic::types::tuple_version>, long int}; decltype ({anonymous}::pythonic::numpy::concatenate((forward<Types>)(<unnamed>::pythonic::numpy::functor::concatenate::operator()::types)...)) = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
   16 |         return f(std::forward<Types>(types)...);                                                   \
/tmp/tmp6fogbgsc.cpp:48:53:   required from 'typename {anonymous}::__pythran_pythr::try_1::type<argument_type0, argument_type1>::result_type {anonymous}::__pythran_pythr::try_1::operator()(argument_type0, argument_type1) const [with argument_type0 = {anonymous}::pythonic::types::numpy_texpr<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> > >; argument_type1 = {anonymous}::pythonic::types::numpy_texpr<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> > >; typename type<argument_type0, argument_type1>::result_type = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
   48 |       return pythonic::numpy::functor::concatenate{}(pythonic::types::make_tuple(x, y), -1L);
      |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/tmp6fogbgsc.cpp:108:68:   required from here
  108 |                                 auto res = __pythran_pythr::try_1()(x, y);
      |                                            ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
/home/br/miniforge3/envs/scipy-dev/lib/python3.12/site-packages/pythran/pythonic/types/numpy_gexpr.hpp:310:9: error: binding reference of type 'std::remove_cv_t<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> >&>' {aka '{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> >&'} to 'const {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int> >' discards qualifiers
  310 |       : arg(other.arg), slices(other.slices), _shape(other._shape), buffer(other.buffer),
      |         ^~~~~~~~~~~~~~
cc1plus: note: unrecognized command-line option '-Wno-unknown-warning-option' may have been intended to silence earlier diagnostics
WARNING: Compilation error, trying hard to find its origin...
Compilation error, trying hard to find its origin...
CRITICAL: You shall not pass!
E: pythr.py:5:9 error: Invalid argument type for function call to `Callable[[Tuple[int, T128, T129], int], ...]`, no overload found, tried:
Callable[[Iterable[Iterable[Iterable[bool]]]], Array[2d, bool]]
Callable[[Iterable[Iterable[Iterable[complex]]]], Array[2d, complex]]
Callable[[Iterable[Iterable[Iterable[float]]]], Array[2d, float]]
Callable[[Iterable[Iterable[Iterable[int]]]], Array[2d, int]]
Callable[[Iterable[Iterable[bool]]], Array[1d, bool]]
Callable[[Iterable[Iterable[complex]]], Array[1d, complex]]
Callable[[Iterable[Iterable[float]]], Array[1d, float]]
Callable[[Iterable[Iterable[int]]], Array[1d, int]]
Callable[[Tuple[int, T0, T1]], Array[1d, bool]]
Callable[[Tuple[int, T10, T11]], Array[1d, bool]]
Callable[[Tuple[int, T100, T101]], Array[2d, float]]
Callable[[Tuple[int, T102, T103]], Array[2d, float]]
Callable[[Tuple[int, T104, T105]], Array[2d, float]]
Callable[[Tuple[int, T106, T107]], Array[2d, float]]
Callable[[Tuple[int, T108, T109]], Array[2d, float]]
Callable[[Tuple[int, T110, T111]], Array[2d, float]]
Callable[[Tuple[int, T112, T113]], Array[2d, complex]]
Callable[[Tuple[int, T114, T115]], Array[2d, complex]]
Callable[[Tuple[int, T116, T117]], Array[2d, complex]]
Callable[[Tuple[int, T118, T119]], Array[2d, complex]]
Callable[[Tuple[int, T12, T13]], Array[1d, bool]]
Callable[[Tuple[int, T120, T121]], Array[2d, complex]]
Callable[[Tuple[int, T122, T123]], Array[2d, complex]]
Callable[[Tuple[int, T124, T125]], Array[2d, complex]]
Callable[[Tuple[int, T126, T127]], Array[2d, complex]]
Callable[[Tuple[int, T14, T15]], Array[1d, bool]]
Callable[[Tuple[int, T16, T17]], Array[1d, int]]
Callable[[Tuple[int, T18, T19]], Array[1d, int]]
Callable[[Tuple[int, T2, T3]], Array[1d, bool]]
Callable[[Tuple[int, T20, T21]], Array[1d, int]]
Callable[[Tuple[int, T22, T23]], Array[1d, int]]
Callable[[Tuple[int, T24, T25]], Array[1d, int]]
Callable[[Tuple[int, T26, T27]], Array[1d, int]]
Callable[[Tuple[int, T28, T29]], Array[1d, int]]
Callable[[Tuple[int, T30, T31]], Array[1d, int]]
Callable[[Tuple[int, T32, T33]], Array[1d, float]]
Callable[[Tuple[int, T34, T35]], Array[1d, float]]
Callable[[Tuple[int, T36, T37]], Array[1d, float]]
Callable[[Tuple[int, T38, T39]], Array[1d, float]]
Callable[[Tuple[int, T4, T5]], Array[1d, bool]]
Callable[[Tuple[int, T40, T41]], Array[1d, float]]
Callable[[Tuple[int, T42, T43]], Array[1d, float]]
Callable[[Tuple[int, T44, T45]], Array[1d, float]]
Callable[[Tuple[int, T46, T47]], Array[1d, float]]
Callable[[Tuple[int, T48, T49]], Array[1d, complex]]
Callable[[Tuple[int, T50, T51]], Array[1d, complex]]
Callable[[Tuple[int, T52, T53]], Array[1d, complex]]
Callable[[Tuple[int, T54, T55]], Array[1d, complex]]
Callable[[Tuple[int, T56, T57]], Array[1d, complex]]
Callable[[Tuple[int, T58, T59]], Array[1d, complex]]
Callable[[Tuple[int, T6, T7]], Array[1d, bool]]
Callable[[Tuple[int, T60, T61]], Array[1d, complex]]
Callable[[Tuple[int, T62, T63]], Array[1d, complex]]
Callable[[Tuple[int, T64, T65]], Array[2d, bool]]
Callable[[Tuple[int, T66, T67]], Array[2d, bool]]
Callable[[Tuple[int, T68, T69]], Array[2d, bool]]
Callable[[Tuple[int, T70, T71]], Array[2d, bool]]
Callable[[Tuple[int, T72, T73]], Array[2d, bool]]
Callable[[Tuple[int, T74, T75]], Array[2d, bool]]
Callable[[Tuple[int, T76, T77]], Array[2d, bool]]
Callable[[Tuple[int, T78, T79]], Array[2d, bool]]
Callable[[Tuple[int, T8, T9]], Array[1d, bool]]
Callable[[Tuple[int, T80, T81]], Array[2d, int]]
Callable[[Tuple[int, T82, T83]], Array[2d, int]]
Callable[[Tuple[int, T84, T85]], Array[2d, int]]
Callable[[Tuple[int, T86, T87]], Array[2d, int]]
Callable[[Tuple[int, T88, T89]], Array[2d, int]]
Callable[[Tuple[int, T90, T91]], Array[2d, int]]
Callable[[Tuple[int, T92, T93]], Array[2d, int]]
Callable[[Tuple[int, T94, T95]], Array[2d, int]]
Callable[[Tuple[int, T96, T97]], Array[2d, float]]
Callable[[Tuple[int, T98, T99]], Array[2d, float]]
----
    z2 = np.concatenate((x, y), axis=-1)
         ^~~~ (o_0)
----

You shall not pass!
E: pythr.py:5:9 error: Invalid argument type for function call to `Callable[[Tuple[int, T128, T129], int], ...]`, no overload found, tried:
Callable[[Iterable[Iterable[Iterable[bool]]]], Array[2d, bool]]
Callable[[Iterable[Iterable[Iterable[complex]]]], Array[2d, complex]]
Callable[[Iterable[Iterable[Iterable[float]]]], Array[2d, float]]
Callable[[Iterable[Iterable[Iterable[int]]]], Array[2d, int]]
Callable[[Iterable[Iterable[bool]]], Array[1d, bool]]
Callable[[Iterable[Iterable[complex]]], Array[1d, complex]]
Callable[[Iterable[Iterable[float]]], Array[1d, float]]
Callable[[Iterable[Iterable[int]]], Array[1d, int]]
Callable[[Tuple[int, T0, T1]], Array[1d, bool]]
Callable[[Tuple[int, T10, T11]], Array[1d, bool]]
Callable[[Tuple[int, T100, T101]], Array[2d, float]]
Callable[[Tuple[int, T102, T103]], Array[2d, float]]
Callable[[Tuple[int, T104, T105]], Array[2d, float]]
Callable[[Tuple[int, T106, T107]], Array[2d, float]]
Callable[[Tuple[int, T108, T109]], Array[2d, float]]
Callable[[Tuple[int, T110, T111]], Array[2d, float]]
Callable[[Tuple[int, T112, T113]], Array[2d, complex]]
Callable[[Tuple[int, T114, T115]], Array[2d, complex]]
Callable[[Tuple[int, T116, T117]], Array[2d, complex]]
Callable[[Tuple[int, T118, T119]], Array[2d, complex]]
Callable[[Tuple[int, T12, T13]], Array[1d, bool]]
Callable[[Tuple[int, T120, T121]], Array[2d, complex]]
Callable[[Tuple[int, T122, T123]], Array[2d, complex]]
Callable[[Tuple[int, T124, T125]], Array[2d, complex]]
Callable[[Tuple[int, T126, T127]], Array[2d, complex]]
Callable[[Tuple[int, T14, T15]], Array[1d, bool]]
Callable[[Tuple[int, T16, T17]], Array[1d, int]]
Callable[[Tuple[int, T18, T19]], Array[1d, int]]
Callable[[Tuple[int, T2, T3]], Array[1d, bool]]
Callable[[Tuple[int, T20, T21]], Array[1d, int]]
Callable[[Tuple[int, T22, T23]], Array[1d, int]]
Callable[[Tuple[int, T24, T25]], Array[1d, int]]
Callable[[Tuple[int, T26, T27]], Array[1d, int]]
Callable[[Tuple[int, T28, T29]], Array[1d, int]]
Callable[[Tuple[int, T30, T31]], Array[1d, int]]
Callable[[Tuple[int, T32, T33]], Array[1d, float]]
Callable[[Tuple[int, T34, T35]], Array[1d, float]]
Callable[[Tuple[int, T36, T37]], Array[1d, float]]
Callable[[Tuple[int, T38, T39]], Array[1d, float]]
Callable[[Tuple[int, T4, T5]], Array[1d, bool]]
Callable[[Tuple[int, T40, T41]], Array[1d, float]]
Callable[[Tuple[int, T42, T43]], Array[1d, float]]
Callable[[Tuple[int, T44, T45]], Array[1d, float]]
Callable[[Tuple[int, T46, T47]], Array[1d, float]]
Callable[[Tuple[int, T48, T49]], Array[1d, complex]]
Callable[[Tuple[int, T50, T51]], Array[1d, complex]]
Callable[[Tuple[int, T52, T53]], Array[1d, complex]]
Callable[[Tuple[int, T54, T55]], Array[1d, complex]]
Callable[[Tuple[int, T56, T57]], Array[1d, complex]]
Callable[[Tuple[int, T58, T59]], Array[1d, complex]]
Callable[[Tuple[int, T6, T7]], Array[1d, bool]]
Callable[[Tuple[int, T60, T61]], Array[1d, complex]]
Callable[[Tuple[int, T62, T63]], Array[1d, complex]]
Callable[[Tuple[int, T64, T65]], Array[2d, bool]]
Callable[[Tuple[int, T66, T67]], Array[2d, bool]]
Callable[[Tuple[int, T68, T69]], Array[2d, bool]]
Callable[[Tuple[int, T70, T71]], Array[2d, bool]]
Callable[[Tuple[int, T72, T73]], Array[2d, bool]]
Callable[[Tuple[int, T74, T75]], Array[2d, bool]]
Callable[[Tuple[int, T76, T77]], Array[2d, bool]]
Callable[[Tuple[int, T78, T79]], Array[2d, bool]]
Callable[[Tuple[int, T8, T9]], Array[1d, bool]]
Callable[[Tuple[int, T80, T81]], Array[2d, int]]
Callable[[Tuple[int, T82, T83]], Array[2d, int]]
Callable[[Tuple[int, T84, T85]], Array[2d, int]]
Callable[[Tuple[int, T86, T87]], Array[2d, int]]
Callable[[Tuple[int, T88, T89]], Array[2d, int]]
Callable[[Tuple[int, T90, T91]], Array[2d, int]]
Callable[[Tuple[int, T92, T93]], Array[2d, int]]
Callable[[Tuple[int, T94, T95]], Array[2d, int]]
Callable[[Tuple[int, T96, T97]], Array[2d, float]]
Callable[[Tuple[int, T98, T99]], Array[2d, float]]
----
    z2 = np.concatenate((x, y), axis=-1)
         ^~~~ (o_0)
----

ev-br · 2025-12-15T15:58:02Z

Oh, I see the same error with

$ pythran --version
0.18.0

so it's not specific to this branch. Now I'm seriously confused --- how come it compiles code which promemently features xp.concat. Which is where this try_1 comes, as a debug implement for a larger user code:

    vec = xp.concat(
        [
            kernel_func(
                xp.linalg.vector_norm(
                    xeps[:, None, :] - yeps[None, :, :], axis=-1
                ), xp
            ),
            xp.prod(xhat[:, None, :] ** powers, axis=-1)
        ], axis=-1
    )

    return vec

EDIT: since it seems to be a pre-existing issue / a feature request, please let me know if you want me to take it to a separate issue.

serge-sans-paille · 2025-12-15T21:34:20Z

Oh, I see the same error with
$ pythran --version
0.18.0
so it's not specific to this branch. Now I'm seriously confused --- how come it compiles code which promemently features xp.concat. Which is where this try_1 comes, as a debug implement for a larger user code:
    vec = xp.concat(
        [
            kernel_func(
                xp.linalg.vector_norm(
                    xeps[:, None, :] - yeps[None, :, :], axis=-1
                ), xp
            ),
            xp.prod(xhat[:, None, :] ** powers, axis=-1)
        ], axis=-1
    )

    return vec
EDIT: since it seems to be a pre-existing issue / a feature request, please let me know if you want me to take it to a separate issue.

#2381 should do the trick (the implementation is a bit slow though, I need to investigate that particular aspect)

Using serge-sans-paille/pythran#2371

ev-br · 2025-12-20T09:29:48Z

I'm happy to confirm that #2381 did the trick! With

(scipy-dev) br@gonzales:~/repos/scipy/scipy$ pushd ~/repos/pythran/
~/repos/pythran ~/repos/scipy/scipy
(scipy-dev) br@gonzales:~/repos/pythran$ git slog
971e7adcc (HEAD -> bug/2367) Merge remote-tracking branch 'origin/fix/concat' into bug/2367
3740eeac8 (origin/fix/concat) Fix np.concatenate when receiving negative axis and/or matrices
61ee9d21b (origin/bug/2367) fix-doc-test
3cef58bb6 Support passing package name as pythran signature
0a139e59f Fix specification error reporting
abdfcc6c8 (origin/master, origin/HEAD) Add support for numpy.concat, numpy.linalg.vector_norm and numpy.linalg.matrix_norm
561fcc7c8 Silent in valid gcc warning in pythonic/numpy/median.hpp
369012115 Update gast and beniget requirements

scipy builds and passes (most) tests. There are several small hiccups I need to fix, and then rerun benchmarks. The ball is squarely in my court. Thank you!

serge-sans-paille · 2025-12-20T11:24:31Z

#2381 merged then, even though I'll have to improve performance for it later. I'll update this PR, improve it slightly (mostly testing for regression etc) then merge it, if that's fine with you?

We must keep exactly the same character count for it to work correctly.

ev-br · 2025-12-20T11:54:29Z

Thank you for being so responsive! A direct channel to the dev is priceless for a user.

Your plan is totally fine for me! Performance will be a concern, definitely (us, we hates them perf regressionses). I'll report what I'll find for the scipy usage. Here or elsewhere? Whichever is more convenient.

serge-sans-paille · 2025-12-20T12:31:00Z

Thanks for the kind words 🙇 Reporting here is fine.

ev-br · 2025-12-21T08:56:52Z

Okay, first crude benchmark: the user code which motivated this feature request is about 2x slower.

While the following is certainly out of scope for this feature (which is great regardless), I'll ask nonetheless. Mainly to gauge if what I observe is fixable or not at all. And if it is fixable, roughly what it needs from the compiler and from the downstream dev (me).

Basically, the change is from the style of (https://github.com/scipy/scipy/blob/v1.16.2/scipy/interpolate/_rbfinterp_pythran.py#L210)

    vec = np.empty((q, p + r), dtype=float)
    for i in range(q):
        vec[i] = ....

to (https://github.com/ev-br/scipy/blob/rbf_xp_dedupe/scipy/interpolate/_rbfinterp_pythran_src.py#L197)

  vec = np.concatenate([ np.linalg.norm(3d_array, axis=-1), other_2d_array  ], axis=-1)

So instead of filling a preallocated array, it relies on numpy vectorized constructs.

In case a profile is useful, here's one:

So questions:

any chance it's related to slowdowns you mentioned in Fix np.concatenate when receiving negative axis and/or matrices #2381 ?
any chance the performance of "vectorized" code can be made on par with the "preallocate-and-fill" code? If yes, what would it take?
any tweaks I can take as a user (inline np.linalg.norm?) while keeping the "vectorized" code style?

Like I was saying, the feature is great regardless of whether this particular case takes a prohibitive perf hit.

serge-sans-paille · 2025-12-21T22:02:54Z

How strange:

$ cat t.py
import numpy as np
#pythran export r(float64[:,:,:], float64[:,:])
def r(a_3d_array, other_2d_array):
    return np.concatenate((np.linalg.norm(a_3d_array, axis=-1), other_2d_array), axis=-1)
$ python -m timeit -s 'import numpy as np; from t import r; x = np.ones((100,100,100)) ; y = np.ones((100,100))' 'r(x, y)'
500 loops, best of 5: 871 usec per loop
$ pythran t.py
$ python -m timeit -s 'import numpy as np; from t import r; x = np.ones((100,100,100)) ; y = np.ones((100,100))' 'r(x, y)'
500 loops, best of 5: 596 usec per loop

but maybe you have different kind of input parameter?

This patch introduces a new argument type of the form `<name> pkg`, for instance `numpy pkg`. It's a special function export type which, if used, must be used consistently across all overloads for that exported function. To support that change, a new pre-processing step has been added. It performs package inference, and tries hard to statically resolve a package name based on the lookup made on it. If a single package matches the constraints, that package is picked. Otherwise nothing is done. The pythran signature just adds an extra constraint. For instance, #pythran export foo(float, numpy pkg) def foo(x, np): return np.cos(x) is equivalent to the following: #pythran export foo(float, numpy pkg) def foo(x, _): import numpy as np return np.cos(x) The exported foo function still has two parameters and the second parameter is going to check that the passed argument is a module named 'numpy'. Fix #2367

serge-sans-paille force-pushed the bug/2367 branch from c9b1075 to 9bc3cdf Compare November 15, 2025 23:31

serge-sans-paille force-pushed the bug/2367 branch from 9bc3cdf to e7e4918 Compare November 16, 2025 13:22

serge-sans-paille force-pushed the bug/2367 branch from e7e4918 to d9e3913 Compare November 16, 2025 13:23

serge-sans-paille force-pushed the bug/2367 branch from d9e3913 to 08c5c4d Compare November 30, 2025 19:40

serge-sans-paille mentioned this pull request Nov 30, 2025

Allow annotating a function argument as numpy #2367

Open

serge-sans-paille force-pushed the bug/2367 branch 2 times, most recently from a8997b6 to 490caf6 Compare December 1, 2025 21:21

ev-br added a commit to ev-br/scipy that referenced this pull request Dec 2, 2025

WIP: start deduplicating pythran / "generic" RBF infra

4d964d2

Using serge-sans-paille/pythran#2371

serge-sans-paille force-pushed the bug/2367 branch from a4bc83a to fcd10e4 Compare December 7, 2025 14:30

serge-sans-paille force-pushed the bug/2367 branch 2 times, most recently from 2fb26de to 261d434 Compare December 8, 2025 09:52

ev-br mentioned this pull request Dec 8, 2025

Recongnize numpy 2.0 / array API aliases #2379

Closed

serge-sans-paille force-pushed the bug/2367 branch from 261d434 to 70d1b66 Compare December 9, 2025 17:04

ev-br added a commit to ev-br/scipy that referenced this pull request Dec 14, 2025

WIP: start deduplicating pythran / "generic" RBF infra

bd507df

Using serge-sans-paille/pythran#2371

serge-sans-paille force-pushed the bug/2367 branch 2 times, most recently from 32d2205 to 01b7582 Compare December 14, 2025 14:55

serge-sans-paille force-pushed the bug/2367 branch from 01b7582 to a971d4c Compare December 14, 2025 15:20

serge-sans-paille force-pushed the bug/2367 branch 2 times, most recently from 1065ebe to 2201042 Compare December 14, 2025 18:18

serge-sans-paille force-pushed the bug/2367 branch from 2201042 to d8dee70 Compare December 14, 2025 21:12

serge-sans-paille force-pushed the bug/2367 branch from d8dee70 to 278a39e Compare December 15, 2025 12:10

ev-br mentioned this pull request Dec 16, 2025

Fix np.concatenate when receiving negative axis and/or matrices #2381

Merged

serge-sans-paille force-pushed the bug/2367 branch from 278a39e to 61ee9d2 Compare December 16, 2025 10:53

ev-br added a commit to ev-br/scipy that referenced this pull request Dec 20, 2025

WIP: start deduplicating pythran / "generic" RBF infra

9346c6f

Using serge-sans-paille/pythran#2371

Fix specification error reporting

6129a42

We must keep exactly the same character count for it to work correctly.

serge-sans-paille force-pushed the bug/2367 branch from 61ee9d2 to 519308d Compare December 20, 2025 11:25

serge-sans-paille force-pushed the bug/2367 branch from 519308d to 060e174 Compare December 21, 2025 22:35

[WIP] Support passing package name as pythran signature #2371

Are you sure you want to change the base?

[WIP] Support passing package name as pythran signature #2371

Uh oh!

Conversation

serge-sans-paille commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serge-sans-paille commented Nov 15, 2025

Uh oh!

codspeed-hq bot commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #2371 will not alter performance

Summary

Uh oh!

ev-br commented Nov 16, 2025

Uh oh!

serge-sans-paille commented Nov 16, 2025

Uh oh!

serge-sans-paille commented Nov 16, 2025

Uh oh!

ev-br commented Nov 16, 2025

Uh oh!

serge-sans-paille commented Nov 16, 2025

Uh oh!

serge-sans-paille commented Dec 7, 2025

Uh oh!

ev-br commented Dec 8, 2025

Uh oh!

serge-sans-paille commented Dec 9, 2025

Uh oh!

serge-sans-paille commented Dec 12, 2025

Uh oh!

ev-br commented Dec 14, 2025

Uh oh!

serge-sans-paille commented Dec 14, 2025

Uh oh!

ev-br commented Dec 14, 2025

Uh oh!

ev-br commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serge-sans-paille commented Dec 14, 2025

Uh oh!

ev-br commented Dec 14, 2025

Uh oh!

ev-br commented Dec 15, 2025

Uh oh!

serge-sans-paille commented Dec 15, 2025 via email

Uh oh!

ev-br commented Dec 15, 2025

Uh oh!

serge-sans-paille commented Dec 15, 2025 via email

Uh oh!

ev-br commented Dec 15, 2025

Uh oh!

ev-br commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serge-sans-paille commented Dec 15, 2025

Uh oh!

ev-br commented Dec 20, 2025

Uh oh!

serge-sans-paille commented Dec 20, 2025

Uh oh!

ev-br commented Dec 20, 2025

Uh oh!

serge-sans-paille commented Dec 20, 2025 via email

Uh oh!

ev-br commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serge-sans-paille commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

serge-sans-paille commented Nov 15, 2025 •

edited

Loading

codspeed-hq bot commented Nov 16, 2025 •

edited

Loading

ev-br commented Dec 14, 2025 •

edited

Loading

ev-br commented Dec 15, 2025 •

edited

Loading

ev-br commented Dec 21, 2025 •

edited

Loading