Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docu for deployers and a simple example #2946

Merged
merged 21 commits into from
Feb 6, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5f8e71e
Add deployer documentation and simple example
AbrilRBS Feb 2, 2023
8220988
Fix note formatting and typo
AbrilRBS Feb 2, 2023
43c14b1
Use root conanfile for copy
AbrilRBS Feb 2, 2023
d71b4bb
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
5f5c892
Merge branch 'conan-io:release/2.0.0-beta' into release/2.0.0-beta
AbrilRBS Feb 6, 2023
721372a
Address review comments
AbrilRBS Feb 6, 2023
c6f43d3
Extra changes
AbrilRBS Feb 6, 2023
72ed52a
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
0aaecd0
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
1b1363f
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
9d501ef
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
12fac7a
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
80ed7f1
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
26da31a
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
f3a31e1
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
21f66b4
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
faf1365
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
27448c7
Update examples/extensions/deployers/sources/custom_deployer_sources.rst
AbrilRBS Feb 6, 2023
f774c5e
Update reference/extensions/deployers.rst
AbrilRBS Feb 6, 2023
81c0993
Remove fn parenthesis to full deployers
AbrilRBS Feb 6, 2023
125bba4
Update reference/extensions/deployers.rst
franramirez688 Feb 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Examples

examples/conanfile/conanfile
examples/extensions/commands/custom_commands
examples/extensions/deployers/custom_deployers
examples/tools/cmake/cmake
examples/tools/files/files
examples/tools/meson/meson
Expand Down
14 changes: 14 additions & 0 deletions examples/extensions/deployers/custom_deployers.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. _examples_extensions_deployers:




Custom deployers
================


.. toctree::
:maxdepth: 2


sources/custom_deployer_sources
80 changes: 80 additions & 0 deletions examples/extensions/deployers/sources/custom_deployer_sources.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
.. examples_extensions_deployers_sources:

Copy sources from all your dependencies
=======================================



Please, first of all, clone the sources to recreate this project. You can find them in the
`examples2.0 repository <https://github.com/conan-io/examples2>`_ in GitHub:
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: bash

$ git clone https://github.com/conan-io/examples2.git
$ cd examples2/examples/extensions/deployers/sources
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get:

cd: no such file or directory: examples2/examples/extensions/deployers/sources

is it expected? or typo in path?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is expected as the PR in examples2 has not yet been merged



In this example we are going to see how to create and use a custom deployer.
This deployer copies all the source files from your dependencies and puts them in a specific output folder
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

.. note::

To better understand this example, it is highly recommended to have previously read the :ref:`Deployers <reference_extensions_deployer_direct_deploy>` reference.


Locate the deployer
-------------------

In this case, the deployer is located in the same directory than our example conanfile,
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
but as show in :ref:`Deployers <reference_extensions_deployer_direct_deploy>` reference,
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
Conan will look for the specified deployer in a few extra places in order, namely:

#. Absolute paths
#. Relative to cwd
#. In the ``[CONAN_HOME]/extensions/deploy`` folder
#. As built-in deployers
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved


Run it
------

For our example, we have a simple recipe that only lists ``zlib`` as a requirement.
With the help of the ``tools.build:download_source=True`` conf, we can force the invocation of its ``source()`` method,
which will ensure that sources are available even if no build needs to be carried out.

Now, you should be able to use the new deployer in both ``conan install`` and ``conan graph`` commands for any given recipe:

.. code-block:: bash

$ conan graph info . -c tools.build:download_source=True --deploy=sources_deploy
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved



Inspecting the command output we can see that it copied the sources of all our dependencies (direct and transitive)
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
to a ``dependency_sources`` folder. After this, extra preprocessing could be accomplished to more specific needs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dependencies_sources might be better?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this dependency_sources folder located?


Code tour
---------

The **source_deploy.py** file has the following code:


**sources_deploy.py**
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: python
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

from conan.tools.files import copy
import os
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, as it's for conan 2 and python 3+, it may already use Pathlib



def deploy(graph, output_folder):
for name, dep in graph.root.conanfile.dependencies.items():
copy(graph.root.conanfile, "*", dep.folders.source_folder, os.path.join(output_folder, "dependency_sources", str(dep)))


deploy()
++++++++

The ``deploy()`` method is called by Conan, and gets both a dependency graph and an output folder path as arguments.
It iterates all the dependencies of our recipe, and copies every source file to their respective folders
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
under ``dependency_sources`` using :ref:`conan.tools.copy<conan_tools_files_copy>`.
73 changes: 72 additions & 1 deletion reference/extensions/deployers.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,76 @@
.. _reference_extensions_deployers:

Deployers
---------
=========

Deployers are a mechanism to facilitate copying files form one folder, usually the Conan cache, to user folders.
While Conan provides two built-in ones (``full_deploy`` and ``direct_deploy``), users can easily manage their own
with ``conan config install``.

Deployers run before generators, and they can change the target folders.
For example, if the ``--deploy=full_deploy`` deployer runs before ``CMakeDeps``,
the files generated by ``CMakeDeps`` will point to the local copy in the user folder done by the ``full_deploy`` deployer,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do they need to affect on generators behavior? that's super confusing part.
as a consumer, I'd expect deployers just to deploy (copy) things, and otherwise, leave other parts untouched.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is one of most requested behaviors for deployers and generators.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's kinda surprising behavior that deployers may modify files generated by generators. can it be made more explicit? e.g. --deploy=full_deploy --deploy-modify-generator=CMakeDeps?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is kinda unspecified, what happens in case of multiple deployers, if all of them want to change target directories, what happens?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, the last one would win

and not to the Conan cache. Multiple deployers can be specified by supplying more than one ``--deploy=`` argument,
and they will be ran in order of appearance.

Deployers can be multi-configuration. Running ``conan install . --deploy=full_deploy`` repeatedly for different profiles
can achieve a fully self-contained project, including all the artifacts, binaries, and build files.
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
This project would be completely independent of Conan and no longer require it at all to build.
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved


Built-in deployers
------------------

.. _reference_extensions_deployer_full_deploy:

full_deploy()
^^^^^^^^^^^^^

Deploys each package folder of every dependency to your recipe's ``output_folder`` in a subfolder tree based on:

#. The build context
#. The dependency name and version
#. The build type
#. The build arch

Such that every dependency will end up in a folder such as:
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

``[OUTPUT_FOLDER]/host/dep/0.1/Release/x86_64``
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as it doesn't include all the settings/options, is there a risk that several deployments will overwrite each other? maybe it's better just to use package id here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is, but if the user is worried about such problems, they are encouraged to write their own deployer, but let's tag @memsharded in case he thinks this could be nice to have



.. _reference_extensions_deployer_direct_deploy:

direct_deploy()
^^^^^^^^^^^^^^^

Same as ``full_deploy()```, but only processes your recipe's *direct* dependencies.
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved


Custom deployers
----------------

Custom deployers can be managed via ``conan config install``. When looking for a specific deployer,
Conan will look in these locations for the deployer in the following order:

#. Absolute paths
#. Relative to cwd
#. In the ``[CONAN_HOME]/extensions/deploy`` folder
#. As built-in deployers

For each installed file, Conan will look for a ``deploy()`` method which to call.
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved
The function signature of your custom deployers should be as follows:

**my_custom_deployer.py**
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: python
AbrilRBS marked this conversation as resolved.
Show resolved Hide resolved

def deploy(graph, output_folder: str):

(Note that the arguments are passed as named parameters, so both the ``graph`` and ``output_folder`` names are mandatory)
franramirez688 marked this conversation as resolved.
Show resolved Hide resolved

You can access your conanfile object with ``graph.root.conanfile``.
See :ref:`ConanFile.dependencies<conan_conanfile_model_dependencies>` for information on how to iterate over its dependencies.
Your custom deployer can now be invoked as if it were a built-in deployer using the filename in which it's found,
in this case ``conan install . --deploy=my_custom_deployer``. Note that supplying the **.py** extension is optional.

See the :ref:`custom deployers<examples_extensions_deployers>` section for examples on how to implement your own deployers.