Building external fortran dependencies using f2py

[KyleAcheson]

I'm having some trouble compiling fortran libraries with f2py. Using the simple example given by the numpy (https://numpy.org/doc/stable/f2py/buildtools/meson.html) meson.build file,

project('f2py_examples', 'c',                                                                                                                                                    
  version : '0.1',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : ['warning_level=2'],
)

add_languages('fortran')

py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

fibby_source = custom_target('fibbymodule.c',
  input : ['fib1.f'],  # .f so no F90 wrappers
  output : ['fibbymodule.c', 'fibby-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'fibby', '--lower']
)

inc_np = include_directories(incdir_numpy, incdir_f2py)

py.extension_module('fibby',
  ['fib1.f', fibby_source],
  incdir_f2py / 'fortranobject.c',
  include_directories: inc_np,
  dependencies : py_dep,
  install : true
)

the code successfully builds,

rm -rf builddir
meson setup builddir
meson compile -C builddir
cd builddir
python -c "import numpy as np; import fibby; a = np.zeros(9); fibby.fib(a); print (a)"
# [ 0.  1.  1.  2.  3.  5.  8. 13. 21.]

However, building a python module that depends on the fortran module fibby using meson-python fails. Now I include the python module fib.py which contains,

import numpy as np                                                                                                                                                               
import fibby                                                                       
                                                                                   
def pyfib(n):                                                                   
    a = np.zeros(n)                                                             
    return fibby.fib(a)

with a pyproject.toml file,

[build-system]                                                                                                                                                                   
build-backend = 'mesonpy'
requires = ['meson-python']

[project]
name = "pyfibby"
version = "0.1.0"
description = """ meson f2py example build"""
requires-python = ">=3.10"
authors = [ 
    {name = "", email = ""},
]
classifiers = []

dependencies = [ 
    "numpy ~=1.26",
]

when running pip install . I get the following error,

Defaulting to user installation because normal site-packages is not writeable
Processing /home/kyle/meson_test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      + meson setup /home/kyle/meson_test /home/kyle/meson_test/.mesonpy-yn6wevis -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/kyle/meson_test/.mesonpy-yn6wevis/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.0
      Source dir: /home/kyle/meson_test
      Build dir: /home/kyle/meson_test/.mesonpy-yn6wevis
      Build type: native build
      Project name: f2py_examples
      Project version: 0.1
      C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      C linker for the host machine: cc ld.bfd 2.38
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      ../meson.build:8: WARNING: add_languages is missing native:, assuming languages are wanted for both host and build.
      Fortran compiler for the host machine: gfortran (gcc 11.4.0 "GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
      Fortran linker for the host machine: gfortran ld.bfd 2.38
      Program python found: YES (/opt/miniconda3/bin/python)
      Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
      Run-time dependency python found: YES 3.10
      
      ../meson.build:14:15: ERROR: Command `/opt/miniconda3/bin/python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'` failed with status 1.
      
      A full log can be found at /home/kyle/meson_test/.mesonpy-yn6wevis/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

which results from the command /opt/miniconda3/bin/python -c 'import os; os.chdir(".."); import numpy; print(numpy.get_include())' in meson.build throwing an error that numpy does not exist despite it being installed and the fact that running this command in the terminal returns the path /home/kyle/.local/lib/python3.10/site-packages/numpy/core/include. I am new to the meson-python backed and honestly have no idea how to debug this, any help would be great. I would eventually like to build a more complicated project that has many fortran dependencies and that can just be installed with pip install .

[eli-schwartz]

Pip uses "build isolation" by default.

You need numpy as a build-system.requires, not just as an install-time dependency, because it needs to be available at build time to run f2py. You're currently missing that, though. The build isolation means even if you have a preinstalled copy of numpy, the build process doesn't see it -- it is built inside a fresh, empty virtualenv.

Note: this isn't really specific to a build backend.

[KyleAcheson]

Thanks, having added the install-time dependency and moving both the fortran source file fib1.f and the python module fib.py out of the project root and into a subdir pyfibby it installs the .so file in ~/.local/lib/python3.10/site-packages/fibby.cpython-310-x86_64-linux-gnu.so and creates ~/.local/lib/python3.10/site-packages/pyfibby-0.1.0.dist-info/.

However, trying to import pyfibby gives me a module not found error, while I can import the fortran library alone and directly access the fortran routine by importing fibby,

>>> import pyfibby
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyfibby'
>>> import fibby
>>> fibby.fib
<fortran function fib>

My meson.build now looks like,

project('f2py_examples', 'c',                                                                                                                                                    
  version : '0.1',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : ['warning_level=2'],
)

add_languages('fortran')

py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

fibby_source = custom_target('fibbymodule.c',
  input : ['pyfibby/fib1.f'],  # .f so no F90 wrappers
  output : ['fibbymodule.c', 'fibby-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'fibby', '--lower']
)

inc_np = include_directories(incdir_numpy, incdir_f2py)

py.extension_module('fibby',
  ['pyfibby/fib1.f', fibby_source],
  incdir_f2py / 'fortranobject.c',
  include_directories: inc_np,
  dependencies : py_dep,
  install : true
)

I wish to be able to import the python wrapper to the fortran routine as pyfibby.fib.pyfib while the underlying fortran library is not directly user accessible. I am guessing that the .so file should not be in the top level of site-packages/ and should be written to site-packages/pyfibby/ (which does not exist) and that these need to be accessible to the python module installed there. However, I am not sure how to do this and I can't seem to find many examples of how to build python modules dependent on fortran in the docs - if you could point me to the right place to look that would be great.

[rgommers]

You should modify the install directory for the py.extension_module call to include 'fibby', and also install an __init__.py file under fibby/ with py.install_sources. That should fix it I think.

[pmaxted]

Thanks for this question and answers - it is the most useful thing I have found so far to help me migrate from distutils to meson.

I did not understand the comment "You should modify the install directory for the py.extension_module call to include 'fibby'", but it does not seem to matter.

I did create an empty __init__.py file in the subdirectory pyfibby.

For everyone else wanting to use this example as a starting point, here are the final versions of all the files that worked for me ...

meson.build

project('f2py_examples', 'c',
  version : '0.1',
  license: 'BSD-3',
  meson_version: '>=0.64.0',
  default_options : ['warning_level=2'],
)

add_languages('fortran')

py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

fibby_source = custom_target('fibbymodule.c',
  input : ['pyfibby/fib1.f'],  # .f so no F90 wrappers
  output : ['fibbymodule.c', 'fibby-f2pywrappers.f'],
  command : [py, '-m', 'numpy.f2py', '@INPUT@', '-m', 'fibby', '--lower']
)

inc_np = include_directories(incdir_numpy, incdir_f2py)

py.extension_module('fibby',
  ['pyfibby/fib1.f', fibby_source],
  incdir_f2py / 'fortranobject.c',
  include_directories: inc_np,
  dependencies : py_dep,
  install : true
)

pyproject.toml

[build-system]
build-backend = 'mesonpy'
requires = ['meson-python','numpy']

[project]
name = "pyfibby"
version = "0.1.0"
description = """ meson f2py example build"""
requires-python = ">=3.10"
authors = [
    {name = "", email = ""},
]
classifiers = []

dependencies = [
    "numpy ~=1.26",
]

pyfibby/fib1.f

C FILE: FIB1.F
      SUBROUTINE FIB(A,N)
C
C     CALCULATE FIRST N FIBONACCI NUMBERS
C
      INTEGER N
      REAL*8 A(N)
      DO I=1,N
         IF (I.EQ.1) THEN
            A(I) = 0.0D0
         ELSEIF (I.EQ.2) THEN
            A(I) = 1.0D0
         ELSE
            A(I) = A(I-1) + A(I-2)
         ENDIF
      ENDDO
      END
C END FILE FIB1.F

pyfibby/fib.py

import numpy as np
import fibby
def pyfib(n):
    a = np.zeros(n)
    return fibby.fib(a)

P.S. remember to pip install meson and pip install ninja before trying this example. Is there some way to make pip install these for itself?