Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] finalize_distribution_options #3447

Closed
gentlegiantJGC opened this issue Jul 13, 2022 · 6 comments
Closed

[BUG] finalize_distribution_options #3447

gentlegiantJGC opened this issue Jul 13, 2022 · 6 comments
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.

Comments

@gentlegiantJGC
Copy link
Contributor

setuptools version

63.1.0

Python version

3.9.13

OS

Windows

Additional environment information

No response

Description

I am trying to implement a sub-command to extend setuptools to add a compile step however this is only run if the local path is added to the python path. This may be an issue in conjunction with the build library.

I am trying to implement this behaviour as was linked to in #2899

I have defined the function in the setup.cfg file however this is only run if the local path is added to sys.path in the setup.py file

[options.entry_points]
setuptools.finalize_distribution_options =
    compile_example_files = my_package.finalise:install

When running python -m build that function is not run.

If the following line is added to the setup.py file the function will get called and everything will run as expected.
sys.path.append(os.path.dirname(__name__))

Sorry if there is something I am misunderstanding here.

Expected behavior

finalize_distribution_options should be able to work with build without modifying the python path.

How to Reproduce

  1. Extract the attached zip file
  2. Install Python with build
  3. Run python -m build
  4. Note that the command custom_finalise was not run
  5. Uncomment line 5 in the setup.py file
  6. Run python -m build again
  7. Note that custom_finalise was run with the following line being *********** do some things **************

finalize_distribution_options.zip

Output

Console output with sys.path.append(os.path.dirname(__name__)) commented out

(venv) PS D:\finalize_distribution_options> python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 42, wheel)
* Getting dependencies for sdist...
running egg_info
creating my_package.egg-info                                        
writing my_package.egg-info\PKG-INFO                                
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt        
writing top-level names to my_package.egg-info\top_level.txt        
writing manifest file 'my_package.egg-info\SOURCES.txt'             
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'                
writing manifest file 'my_package.egg-info\SOURCES.txt'
* Building sdist...
running sdist
running egg_info                                                    
writing my_package.egg-info\PKG-INFO                                
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
creating my-package-1.0.0
creating my-package-1.0.0\my_package
creating my-package-1.0.0\my_package.egg-info
copying files to my-package-1.0.0...
copying MANIFEST.in -> my-package-1.0.0
copying pyproject.toml -> my-package-1.0.0
copying setup.cfg -> my-package-1.0.0
copying setup.py -> my-package-1.0.0
copying my_package\__init__.py -> my-package-1.0.0\my_package
copying my_package\finalise.py -> my-package-1.0.0\my_package
copying my_package.egg-info\PKG-INFO -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\SOURCES.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\dependency_links.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\entry_points.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\top_level.txt -> my-package-1.0.0\my_package.egg-info
Writing my-package-1.0.0\setup.cfg
Creating tar archive
removing 'my-package-1.0.0' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 42, wheel)
* Getting dependencies for wheel...
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build\lib
creating build\lib\my_package
copying my_package\finalise.py -> build\lib\my_package
copying my_package\__init__.py -> build\lib\my_package
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\my_package
copying build\lib\my_package\finalise.py -> build\bdist.win-amd64\wheel\.\my_package
copying build\lib\my_package\__init__.py -> build\bdist.win-amd64\wheel\.\my_package
running install_egg_info
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
Copying my_package.egg-info to build\bdist.win-amd64\wheel\.\my_package-1.0.0-py3.9.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\my_package-1.0.0.dist-info\WHEEL
creating 'D:\finalize_distribution_options\dist\tmpu9scla6m\my_package-1.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'my_package/__init__.py'
adding 'my_package/finalise.py'
adding 'my_package-1.0.0.dist-info/METADATA'
adding 'my_package-1.0.0.dist-info/WHEEL'
adding 'my_package-1.0.0.dist-info/entry_points.txt'
adding 'my_package-1.0.0.dist-info/top_level.txt'
adding 'my_package-1.0.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
Successfully built my-package-1.0.0.tar.gz and my_package-1.0.0-py3-none-any.whl

Console output after with sys.path.append(os.path.dirname(__name__)) uncommented

(venv) PS D:\finalize_distribution_options> python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 42, wheel)
* Getting dependencies for sdist...
running egg_info
creating my_package.egg-info                                        
writing my_package.egg-info\PKG-INFO                                
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt        
writing top-level names to my_package.egg-info\top_level.txt        
writing manifest file 'my_package.egg-info\SOURCES.txt'             
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'                
writing manifest file 'my_package.egg-info\SOURCES.txt'
* Building sdist...
running sdist
running egg_info                                                    
writing my_package.egg-info\PKG-INFO                                
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
creating my-package-1.0.0
creating my-package-1.0.0\my_package
creating my-package-1.0.0\my_package.egg-info
copying files to my-package-1.0.0...
copying MANIFEST.in -> my-package-1.0.0
copying pyproject.toml -> my-package-1.0.0
copying setup.cfg -> my-package-1.0.0
copying setup.py -> my-package-1.0.0
copying my_package\__init__.py -> my-package-1.0.0\my_package
copying my_package\finalise.py -> my-package-1.0.0\my_package
copying my_package.egg-info\PKG-INFO -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\SOURCES.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\dependency_links.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\entry_points.txt -> my-package-1.0.0\my_package.egg-info
copying my_package.egg-info\top_level.txt -> my-package-1.0.0\my_package.egg-info
Writing my-package-1.0.0\setup.cfg
Creating tar archive
removing 'my-package-1.0.0' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 42, wheel)
* Getting dependencies for wheel...
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build\lib
creating build\lib\my_package
copying my_package\finalise.py -> build\lib\my_package
copying my_package\__init__.py -> build\lib\my_package
running custom_finalise
*********** do some things **************
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\my_package
copying build\lib\my_package\finalise.py -> build\bdist.win-amd64\wheel\.\my_package
copying build\lib\my_package\__init__.py -> build\bdist.win-amd64\wheel\.\my_package
running install_egg_info
running egg_info
writing my_package.egg-info\PKG-INFO
writing dependency_links to my_package.egg-info\dependency_links.txt
writing entry points to my_package.egg-info\entry_points.txt
writing top-level names to my_package.egg-info\top_level.txt
reading manifest file 'my_package.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'my_package.egg-info\SOURCES.txt'
Copying my_package.egg-info to build\bdist.win-amd64\wheel\.\my_package-1.0.0-py3.9.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\my_package-1.0.0.dist-info\WHEEL
creating 'D:\finalize_distribution_options\dist\tmpj63f3pk0\my_package-1.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'my_package/__init__.py'
adding 'my_package/finalise.py'
adding 'my_package-1.0.0.dist-info/METADATA'
adding 'my_package-1.0.0.dist-info/WHEEL'
adding 'my_package-1.0.0.dist-info/entry_points.txt'
adding 'my_package-1.0.0.dist-info/top_level.txt'
adding 'my_package-1.0.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
Successfully built my-package-1.0.0.tar.gz and my_package-1.0.0-py3-none-any.whl
@gentlegiantJGC gentlegiantJGC added bug Needs Triage Issues that need to be evaluated for severity and status. labels Jul 13, 2022
@abravalheri
Copy link
Contributor

Hi @gentlegiantJGC, thank you very much for opening this discussion.

There are 2 facts in play regarding this issue.

  1. The entry-points approach for customizing the build is intended to be used to create setuptools plugins. Setuptools plugins work well after being installed.
    If you want to use your plugin to customise its own build process, then it is a bit more complicated than that.
    Are you trying to create a setuptools plugin or just trying to add build steps to a specific project?

  2. Since v44 (or something similar) Setuptools intentionally avoids adding the current working directory (CWD) to sys.path. Developers that rely on this behaviour need to implement it in setup.py (e.g. sys.path.append(os.path.dirname(__file__))).

Is the project that you are working with opensource? Could I have a look on it?

@gentlegiantJGC
Copy link
Contributor Author

I am trying to add code that runs during the build step to modify the files.

I added a minimal reproducible example as a zip above the console logs in the report but it just prints a line.
I want to use this functionally to merge an unwieldy number of small files into a compressed archive for distribution.

The project I am using this in is here.
https://github.com/gentlegiantJGC/PyMCTranslate/tree/master

@abravalheri
Copy link
Contributor

Hi @gentlegiantJGC thank you very much for the extra info.

I want to use this functionally to merge an unwieldy number of small files into a compressed archive for distribution.

If you want to do that for a single project, maybe instead of finalize_distribution_options I think you could try something different (see the following suggestion).

If you want to do that for multiple projects, than you can use the entry-points approach to create a setuptools plugin.

Suggestion

Let's keep the custom build steps code into a private folder _build_steps. This prevents ad-hoc build command classes to be shared as part of your project's business logic:

.
├── MANIFEST.in
├── _build_steps
│   └── custom_finalise.py
├── my_package
│   └── __init__.py
├── pyproject.toml
├── setup.cfg
└── setup.py

On setup.py we set the stage for the custom build step:

# setup.py
import os
import sys
from setuptools import setup
from setuptools.command.build import build as orig_build

sys.path.append(os.path.join(os.path.dirname(__name__), "_build_steps"))
from custom_finalise import Finalise

class build(orig_build):
    sub_commands = [
        *orig_build.sub_commands,
        ("custom_finalise", None)
    ]

setup(
    cmdclass={"build": build, "custom_finalise": Finalise}
)

Then in _build_steps/custom_finalise.py you can define your custom command:

# _build_steps/custom_finalise.py
from setuptools import Command
from setuptools.dist import Distribution


class Finalise(Command):
    def initialize_options(self):
        pass

    def finalize_options(self):
        pass

    def run(self):
        print("*********** do some things **************")

Don't forget to add this file to the sdist either via setuptools-scm or MANIFEST.in:

recursive-include my_package *.py
recursive-include _build_steps *.py

@gentlegiantJGC
Copy link
Contributor Author

Thanks. That is nicer than the original subclassing system I had.
I am surprised though that there isn't a nicer way to add functionality.

Do you know why finalize_distribution_options works when the root path is added to the python path?
Is that intended behaviour or a bug?

@abravalheri
Copy link
Contributor

Do you know why finalize_distribution_options works when the root path is added to the python path?

If I am not wrong, this happens because importlib_metadata reads the .egg-info directory in all items of sys.path.

I think it is problematic to add the customisation to your project’s own entry-points, right? This would mean that setuptools will try to use the custom build step you define with every single package you try to build in that env...

Is that intended behaviour or a bug?

I don't think it is bug. Setuptools uses this trick to add build-time only entry-points for its own build (you can see in the repository it has a bootstrap.egg-info folder.

@gentlegiantJGC
Copy link
Contributor Author

Okay. Thanks for the help.

I think for this to work in editable mode as well the command would need to be registered as a subcommand of the develop command as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.
Projects
None yet
Development

No branches or pull requests

2 participants