modules: The 'features' module

[seiko2plus]

This module aims to provide a more dynamic way to control common
features between compilers that would generally require special
headers, arguments, or compile-time test cases, quite ideal
to manage CPU features and solve compatibility issues.

TODO:

• [ ] add x86 features
• [ ] add Arm features
• [ ] add PPC64 features
• [ ] add IBMZ features
• add static testing unit
• add document
• improve log/debug
• improve the performance, e.g. LRU

[rgommers]

Looks promising, thanks @seiko2plus!

One question I have is what CPU features are of interest here aside from SIMD ones? From the discussion at #11033 I thought you were going to try and fit this into the existing simd module. I don't personally have a preference here - whatever works and has a good API I'd say.

For others, @seiko2plus posted this test program to try this out earlier (here):

project('test', 'c')

feature = import('feature')
cpu_features = feature.cpu_features()

targets = [['AVX2_FMA3', [cpu_features['AVX2'], cpu_features['FMA3']]],
          ['AVX512_COMMON', cpu_features['AVX512_COMMON']],
          ['AVX512_SKX', cpu_features['AVX512_SKX']]]

foreach target : targets
  t = feature.test(target[1])
  if t[0]
    message('target name:', target[0])
    data = t[1]
    message('Implicit features:', data['implicit'])
    message('Compiler arguments:', data['args'])
    message('Headers:', data['headers'])
    message('Need to detect on runtime:', data['detect'])
    message('Supported #definitions:', data['defines'])
    message('Unsupported #definitions:', data['undefines'])
    message('Compiler support this target by ', data['support'])
    message('========================================')
  endif
endforeach

@seiko2plus this diff fixes the minor issue I had with feature detection here:

--- a/mesonbuild/modules/feature/x86_features.py
+++ b/mesonbuild/modules/feature/x86_features.py
@@ -42,7 +42,7 @@ if T.TYPE_CHECKING:
 
 def test_code_main(code: str) -> str:
     return dedent(f'''\
-        int main(int, char **argv)
+        int main(int unused, char **argv)
         {{
            char *src = argv[1];
            {dedent(code)}

[seiko2plus]

#11033 I thought you were going to try and fit this into the existing simd module.

The current meson SIMD module lacks flexibility and modifies it to support our needs in a way to provide static and dynamic dispatching (baseline/dispatch) that fits "any project needs" will require more effort than I expected.

So I just split the work into two modules rather than a single module covering all the operations, the first module is the module 'feature' which provides what is necessary to be coded by python, and the second module 'compiler_opt' will be left for future step basically it will require to convert the following two meson scripts (not completed yet) into python module:

• CPU dispatcher initializer meson_cpu/meson.build
• Dispatch-able sources builder core/meson_cpu/meson.build

We still can modify the current meson SIMD module to allow only the use of the CPU features in this module without any extra modifications but we're not going to use it in NumPy, for many considerations are hard to explain in one comment.

One question I have is what CPU features are of interest here aside from SIMD ones?

Do you mean aside from SIMD features? Any CPU features in general, but main concern features that will be required for optimizing certain operations. For example, POPCNT and LZCNT.

But if you mean aside from the current meson SIMD module?

Flexibility is the main reason, imagine if a project wants to modify or add a new feature or a new architecture. What if there is a necessity to make fast or custom modifications to an existing feature? some examples:

So assuming adding feature X implies feature AVX512_ICL:

module_feature = import('feature')
cpu_features = module_feature.cpu_features()
icl = cpu_features['AVX512_ICL']
nfet = module_feature.new('X',  icl.get('interest') + 1, implies: icl, test_code='''
// testing code
''')
if cc.get_id() == 'gcc'
   nfet.update(args: '-mX') # flags
else
   nfet.update(disable: 'requires gcc')
endif
cpu_features += {'X': nfet}

What about modifying or adding a certain flag for a certain feature?

We faced an incident here numpy/numpy#20991 which requires disabling MMX registers as a workaround.

if cc.get_id() == 'gcc'
  fet = cpu_features['AVX512_COMMON']
  fet.update(args: fet.get('args') + '-mno-mmx')
endif

Now all the features that imply AVX512_COMMON have the new arg including feature AVX512_COMMON. I will try to cover most of the usage in the document that will come with this module.

[rgommers]

and the second module 'compiler_opt' will be left for future step basically it will require to convert the following two meson scripts (not completed yet) into python module:

I think that'd be a vendorable package containing only meson.build files (and maybe some C/C++ code), which one could put in a subprojects directory, rather than a Python package, right? There is nothing Python-specific about this I think. I'm not sure if there is precedent for that kind of thing in Meson-using projects, but I think it seems like a good plan. At least for now - it's pretty complex, so generalizing it from use by only NumPy to multiple other projects first makes sense to me.

Do you mean aside from SIMD features? Any CPU features in general, but main concern features that will be required for optimizing certain operations. For example, POPCNT and LZCNT.

I meant SIMD features in general. Which is still the main use case here I think. But yes, popcnt and lzcnt is a good example, thanks.

I will try to cover most of the usage in the document that will come with this module.

Nice examples you have given here already, thanks.

[eli-schwartz]

I think that'd be a vendorable package containing only meson.build files (and maybe some C/C++ code), which one could put in a subprojects directory, rather than a Python package, right? There is nothing Python-specific about this I think. I'm not sure if there is precedent for that kind of thing in Meson-using projects, but I think it seems like a good plan. At least for now - it's pretty complex, so generalizing it from use by only NumPy to multiple other projects first makes sense to me.

It certainly would not be the only meson-using project that used a subproject containing sources which are intended strictly for use as a vendorable package, together with the meson.build instructions for building it into a static library and yielding that to the superproject.

[9243659864598]

qggq

[seiko2plus]

@eli-schwartz, Hi Eli, I am curious if there a method to directly generate documentation for the supported methods from the Python scripts themselves.

[eli-schwartz]

@eli-schwartz, Hi Eli, I am curious if there a method to directly generate documentation for the supported methods from the Python scripts themselves.

Historically, no. All the documentation is either handwritten or produced as structured yaml in docs/yaml/. We do have a way to generate module documentation from structured yaml, but it's currently disabled because those were never properly ported (doing this would be nice, but also probably too complicated to hold up the 'feature' module over).

[seiko2plus]

Historically, no. All the documentation is either handwritten or produced as structured yaml in docs/yaml/. We do have a way to generate module documentation from structured yaml, but it's currently disabled because those were never properly ported (doing this would be nice, but also probably too complicated to hold up the 'feature' module over).

Thank you for the clarification, its seem I will go for handwritten but I just realized different formats when it comes to methods signatures and module objects. Could you take a look at current handwritten format that this module use, and suggest format for documenting module objects?

[eli-schwartz]

You could use the python module as partial inspiration: https://mesonbuild.com/Python-module.html#python_installation-object

It is a subheader with a list of methods in further subheaders, which aren't substantially different from module functions.

[eli-schwartz]

A bit of vague high-level commentary:

• there is already a feature (options) object, adding another FeatureObject runs the risk of the two being confused. It's also not entirely clear what an import('features') does. Features of what? I would probably rename the module "cpufeatures".
• In the ModuleInfo, you can pass unstable=True, which will mark the module as provisional. This means that meson reserves the right to change how it works (or in theory even remove it) without falling under the usual semver stability policy. I think it would be a good idea to start off using this, with the understanding that numpy is the primary stakeholder in the module, as it gives us the leeway to rethink the design. Hopefully this won't be necessary -- but it is a pretty complex new module, so you never know. The only practical effect this will have on consumers such as numpy is a) logging a nonfatal warning "btw this is unstable, compatibility not guaranteed", b) if such changes ever do occur, one would need to pay close attention to which versions of meson are allowed by meson.build. Considering you'd effectively be the codeowners in meson for it, this isn't really a burden at all.

[eli-schwartz]

I think this module would also probably merit a tutorial walkthrough, e.g. docs/markdown/Using-CPU-multifunction-dispatch.md.

For bonus points it would compare and contrast both this module and the existing (also unstable) SIMD module. This module is a lot more powerful, so it would be interesting to see how rewriting a SIMD module example would look here -- and by the same token, offer guidance on migrating over to the new module so that we could perhaps deprecate the SIMD module.

[mattip]

Also see #11885 which also proposes a module/features module

[dcbaker]

I don't know that it's import, I can always rename the module in 11885 if we ever have agreement on adding that module, it could be named options for example; and this PR is much closer to landing than that one. So, as the author of 11885, I don't think it's worth worrying about

[bruchar1]

I'd be in favor of renaming it cpufeatures, or maybe simply cpu, just to avoid confusion with the "feature" object that is completly different.

[bruchar1]

symlinks will be problematic on Windows platform.

[tristan957]

Seems like this is pretty close to getting merged. Would be great to get in for 1.6.