ENH: Meson build system support for F2PY's f2py -c command#22225
ENH: Meson build system support for F2PY's f2py -c command#22225NamamiShanker wants to merge 132 commits intonumpy:mainfrom
f2py -c command#22225Conversation
This reverts commit b334020.
| from string import Template | ||
|
|
||
| class MesonTemplate: | ||
| """Template meson build file generation class.""" |
There was a problem hiding this comment.
There's tabs here instead of spaces. @NamamiShanker can you please check your editor settings and ensure tabs are always converted to spaces?
There was a problem hiding this comment.
Sure @rgommers. I corrected this file and the others.
rgommers
left a comment
There was a problem hiding this comment.
Looks like nice progress, thanks @NamamiShanker. I just had a quick look through the build system part of this and made a few comments for things that stood out to me.
| py3 = py_mod.find_installation('python3') | ||
| py3_dep = py3.dependency() | ||
| message(py3.path()) | ||
| message(py3.get_install_dir()) |
There was a problem hiding this comment.
This should have pure: false as argument.
| @@ -0,0 +1,22 @@ | |||
| project('${modulename}', 'c', | |||
| version : '0.1', | |||
There was a problem hiding this comment.
I think you want to set the minimum meson version here, and it should be very recent; lots of Python bugs were fixed recently. I recommend the most recent release, 0.63.2, and bumping it up to 0.63.3/0.64.0 when those become available.
There was a problem hiding this comment.
Corrected it.
| def configuration(parent_package='', top_path=None): | ||
| config = Configuration('f2py', parent_package, top_path) | ||
| config.add_subpackage('tests') | ||
| config.add_subpackage('backends') |
There was a problem hiding this comment.
This is only needed if there's a backends/setup.py.
There was a problem hiding this comment.
I am not much familiar with distutils configuration files. Would it be more apt to use the add_data_dir function instead? With the current configuration, the backend folder will not be included in the installed numpy/f2py folder.
There was a problem hiding this comment.
add_data_dir is more appropriate. However, after looking at how it's done elsewhere, add_subpackage is used more often (even if it gives a warning). So you can leave it as is.
| config = Configuration('f2py', parent_package, top_path) | ||
| config.add_subpackage('tests') | ||
| config.add_subpackage('backends') | ||
| config.add_data_dir('backends/src') |
There was a problem hiding this comment.
I don't think this does anything, the next line covers it already. You should need one or the other I believe.
| "distutils.unixccompiler", | ||
| "dual", | ||
| "f2py.auxfuncs", | ||
| "f2py.backends", |
There was a problem hiding this comment.
This does not look like it should be public, or should it? If not, can you please name it f2py._backends?
There was a problem hiding this comment.
And the same for all the other things - nothing should be added to this list ideally.
| if opt_flags is None: | ||
| opt_flags = [] | ||
| if arch_flags is None: | ||
| arch_flags = [] |
There was a problem hiding this comment.
These flags don't quite fit anymore (the difference between F77 and F90 is not needed, nor are the exact paths to the compiler; the --opt, --arch etc. are specified in a very non-standard way, Meson has a better way to do things like debug flags). Is there a plan to clean this up? Maybe they should be kept unchanged for now, and then when numpy.distutils is gone they can be updated?
|
I am interested in this because I am porting the build system of another package (PySCeS) to try and get rid of the This PR seems to provide just what I need so I tried only compiling the Fortran extension modules on the command line. This is with numpy installed from Using f2py with the default Pertinent versions:
Navigating to the nleq2 source directory, and executing f2py -c nleq2.pyf nleq2.f wnorm.f linalg_nleq2.f zibconst.f zibmon.f zibsec.fworks fine and the *.so binary module file is created, and can be imported from Python. However, executing the following fails: f2py -c --backend meson nleq2.pyf nleq2.f wnorm.f linalg_nleq2.f zibconst.f zibmon.f zibsec.fThe traceback is too big to include inline, I attach it as a file. We use a custom Trying to debug this, I used a custom |
f2py -c commandf2py -c command
|
@HaoZeke @NamamiShanker we're now close to the 1.26.0 release, and this is still pending. I think we do need this for Python 3.12 support of using |
|
I overhauled the random example cython compilation to use meson in #24153, maybe worth seeing if anything there is applicable here. |
|
I'll aim to have this and the previous dependent PR in before the weekend. Should be followed up to ensure most projects start using it (and submit bug reports if needed). |
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
This sounds like a frontend ( |
| def macros_substitution(self) -> None: | ||
| self.substitutions["macros"] = "" | ||
| if self.define_macros: | ||
| self.substitutions["macros"] = ",".join(f"\'-D{macro[0]}={macro[1]}\'" if macro[1] else f"-D{macro[0]}" for macro in self.define_macros) | ||
| if self.undef_macros: | ||
| self.substitutions["macros"] += "," + ",".join(f"\'-U{macro}\'" for macro in self.undef_macros) |
There was a problem hiding this comment.
These don't pass -D flags to the underlying preprocessor, that would have to be via Dc_args=-D{macro[0]}={macro[1]}... (e.g. seen here)
Similar issue with unsetting the defines.
| def global_flags_substitution(self) -> None: | ||
| fortran_compiler_flags = self.fortran_flags + self.optimization_flags + self.architecture_flags | ||
| c_compiler_flags = self.optimization_flags + self.architecture_flags | ||
| self.substitutions["fortran_global_args"] = fortran_compiler_flags | ||
| self.substitutions["c_global_args"] = c_compiler_flags |
There was a problem hiding this comment.
Global flags are generally a bad idea, especially since a lot of these might be used as subprojects, it makes more sense to populate the _project variants.
| self.substitutions['numpy_get_include'] = self.numpy_get_include.absolute() | ||
| self.substitutions['f2py_get_include'] = self.f2py_get_include.absolute() |
There was a problem hiding this comment.
Are these necessary? The documentation includes the option to get these dynamically from meson...
| def linker_substitution(self) -> None: | ||
| self.substitutions["linker_args"] = "" | ||
| if self.linker_libpath: | ||
| linker_libpath_subs = ",".join(f"-L{libpath}" for libpath in self.linker_libpath) | ||
| self.substitutions["linker_args"] += linker_libpath_subs | ||
| if self.linker_libname: | ||
| linker_libname_subs = ",".join(f"-l{libname}" for libname in self.linker_libname) | ||
| self.substitutions["linker_args"] += f",{linker_libname_subs}" |
Works with #24532 , thanks! Traceback (most recent call last):
File "/home/jr/.virtualenvs/numpy-test/bin/f2py", line 33, in <module>
sys.exit(load_entry_point('numpy==2.0.0.dev0', 'console_scripts', 'f2py')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 732, in main
run_compile()
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 705, in run_compile
builder.compile()
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 130, in compile
self.write_meson_build(self.build_dir)
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 108, in write_meson_build
src = meson_template.generate_meson_build()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 71, in generate_meson_build
template = Template(self.meson_build_template())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/_meson.py", line 44, in meson_build_template
raise FileNotFoundError(
FileNotFoundError: [Errno 2] Meson build template /home/jr/.virtualenvs/numpy-test/lib/python3.11/site-packages/numpy/f2py/_backends/meson.build.template does not exist.Looking in the site-packages directory, I noticed that the file |
Should address numpy#22225 (comment)
Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com>
Should address numpy#22225 (comment)
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from gh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from gh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address #22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from numpygh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from numpygh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address numpy#22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from gh-22225 * ENH: Add meson skeleton from gh-2225 * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address #22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Rohit Goswami <rgoswami@quansight.com> Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
|
This is updated in #24532, so closing this. |
* FIX: Import f2py2e rather than f2py for run_main * FIX: Import f2py2e instead of f2py * ENH: Add F2PY back-end work from numpygh-22225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * ENH: Add meson skeleton from numpygh-2225 Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> * MAINT: Trim backend.py down to f2py2e flags * ENH: Add a factory function for backends * ENH: Add a distutils backend * ENH: Handle --backends in f2py Defaults to distutils for now * DOC: Add some minor comments in f2py2e * MAINT: Refactor and rework meson.build.src * MAINT: Add objects * MAINT: Cleanup distutils backend * MAINT: Refactor to add everything back to backend Necessary for the meson.build for now. Refactors / cleanup needs better argument handling in f2py2e * MAINT: Fix overly long line * BUG: Construct wrappers for meson backend * MAINT: Rework, simplify template massively * ENH: Truncate meson.build to skeleton only * MAINT: Minor backend housekeeping, name changes * MAINT: Less absolute paths, update setup.py [f2py] * MAINT: Move f2py module name functionality Previously part of np.distutils * ENH: Handle .pyf files * TST: Fix typo in isoFortranEnvMap.f90 * MAINT: Typo in f2py2e support for pyf files * DOC: Add release note for --backend * MAINT: Conditional switch for Python 3.12 [f2py] * MAINT: No absolute paths in backend [f2py-meson] The files are copied over anyway, this makes it easier to extend the generated skeleton * MAINT: Prettier generated meson.build files [f2py] * ENH: Add meson's dependency(blah) to f2py * DOC: Document the new flag * MAINT: Simplify and rename backend template [f2py] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Support build_type via --debug [f2py-meson] * MAINT,DOC: Reduce warn,rework doc [f2py-meson] Co-authored-by: rgommers <rgommers@users.noreply.github.com> * ENH: Rework deps: to --dep calls [f2py-meson] Also shows how incremental updates to the parser can be done. * MAINT,DOC: Add --backend to argparse, add docs * MAINT: Rename meson template [f2py-meson] * MAINT: Add meson.build for f2py Should address numpy#22225 (comment) * BLD: remove duplicate f2py handling in meson.build files --------- Co-authored-by: Namami Shanker <namami2011@gmail.com> Co-authored-by: NamamiShanker <NamamiShanker@users.noreply.github.com> Co-authored-by: rgommers <rgommers@users.noreply.github.com> Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
This PR is a continuation of my older PR for implementing F2PY's fronted with
argparse.My blog explains the core ideas behind the changes.
This PR introduces a new flag for F2PY -
--backendwhich can accept build system for building Python extension modules.An example to produce a Python module out of the classic fibonacci sequence Fortran program will be:-
The output thus produced works correctly:
The default value of
--backendflag isdistutilsmaking this feature completely backwards compatible.