From d073c6033008f4909a08649ca0ec2d315316c643 Mon Sep 17 00:00:00 2001 From: James Date: Fri, 31 Jul 2020 14:09:48 +0200 Subject: [PATCH] Feature/lockfiles new (#1790) * working * working * working * first draft of intro + multi * Update versioning/lockfiles/introduction.rst Co-authored-by: Carlos Zoido * mising cd .. * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Daniel * Update versioning/lockfiles/configurations.rst Co-authored-by: Daniel * Update versioning/lockfiles/configurations.rst Co-authored-by: Daniel * Update versioning/lockfiles/configurations.rst Co-authored-by: Daniel * Update versioning/lockfiles/configurations.rst Co-authored-by: Daniel * Update versioning/lockfiles/configurations.rst Co-authored-by: Daniel * Update versioning/lockfiles/introduction.rst Co-authored-by: Carlos Zoido * Update versioning/lockfiles/introduction.rst Co-authored-by: Carlos Zoido * build-order * Update versioning/lockfiles/build_order.rst Co-authored-by: Carlos Zoido * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * review * reference * fix broken refs * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * Update versioning/lockfiles/build_order.rst Co-authored-by: Daniel * fix reference indent * clarify full-partial locked packages * image Co-authored-by: Carlos Zoido Co-authored-by: Daniel --- reference/commands.rst | 2 +- reference/commands/consumer/info.rst | 8 +- reference/commands/misc/graph.rst | 217 ------------- reference/commands/misc/lock.rst | 143 +++++++++ reference/commands/output/info.rst | 14 +- versioning/lockfiles.rst | 285 +----------------- versioning/lockfiles/build_order.rst | 216 +++++++++++++ versioning/lockfiles/ci.rst | 10 + .../lockfiles/conan_lock_build_order.png | Bin 0 -> 13121 bytes versioning/lockfiles/configurations.rst | 191 ++++++++++++ versioning/lockfiles/introduction.rst | 258 ++++++++++++++++ 11 files changed, 838 insertions(+), 506 deletions(-) delete mode 100644 reference/commands/misc/graph.rst create mode 100644 reference/commands/misc/lock.rst create mode 100644 versioning/lockfiles/build_order.rst create mode 100644 versioning/lockfiles/ci.rst create mode 100644 versioning/lockfiles/conan_lock_build_order.png create mode 100644 versioning/lockfiles/configurations.rst create mode 100644 versioning/lockfiles/introduction.rst diff --git a/reference/commands.rst b/reference/commands.rst index 85c619a4cec..b516a297121 100644 --- a/reference/commands.rst +++ b/reference/commands.rst @@ -67,7 +67,7 @@ Other useful commands: commands/misc/remove.rst commands/misc/alias.rst commands/misc/inspect.rst - commands/misc/graph.rst + commands/misc/lock.rst commands/misc/help.rst commands/misc/conan_build_info.rst diff --git a/reference/commands/consumer/info.rst b/reference/commands/consumer/info.rst index e09416c82c5..7764cdeeb7f 100644 --- a/reference/commands/consumer/info.rst +++ b/reference/commands/consumer/info.rst @@ -158,12 +158,6 @@ The output will look like: difference is that it doesn't try to install or build the binaries, but the package recipes will be retrieved from remotes if necessary. -.. important:: - - There is a dedicated command to work with the graph of dependencies and to retrieve information - about it. We encourage you to use :ref:`conan graph` instead of this ``conan info`` - command for those tasks. - It is very important to note, that the :command:`info` command outputs the dependency graph for a given configuration (settings, options), as the dependency graph can be different for different configurations. Then, the input to the :command:`conan info` command is the same as :command:`conan install`, @@ -189,7 +183,7 @@ argument: It is possible to use the :command:`conan info` command to extract useful information for Continuous Integration systems. More precisely, it has the :command:`--build-order, -bo` option (deprecated in -favor of :ref:`conan graph build-order`), that will produce +favor of :ref:`conan lock build-order`), that will produce a machine-readable output with an ordered list of package references, in the order they should be built. E.g., let's assume that we have a project that depends on Boost and Poco, which in turn depends on OpenSSL and zlib transitively. So we can query our project with a reference that has diff --git a/reference/commands/misc/graph.rst b/reference/commands/misc/graph.rst deleted file mode 100644 index a633fb09dbd..00000000000 --- a/reference/commands/misc/graph.rst +++ /dev/null @@ -1,217 +0,0 @@ -.. _conan_graph: - -conan graph -=========== - -.. code-block:: bash - - $ conan graph [-h] {update-lock,build-order,clean-modified,lock} ... - -Generates and manipulates lock files. - -.. code-block:: text - - positional arguments: - {update-lock,build-order,clean-modified,lock} - sub-command help - update-lock merge two lockfiles - build-order Returns build-order - clean-modified Clean modified - lock create a lockfile - - optional arguments: - -h, --help show this help message and exit - - - -conan graph update-lock ------------------------ - -.. code-block:: bash - - $ conan graph update-lock [-h] old_lockfile new_lockfile - -Updates the *old_lockfile* file with the contents of the *new_lockfiles*. - -.. code-block:: text - - positional arguments: - old_lockfile path to previous lockfile - new_lockfile path to modified lockfile - - optional arguments: - -h, --help show this help message and exit - - -Only the packages in *new_lockfile* marked as "modified" will be processed. -If a node in *old_lockfile* is already modified and an incompatible (different -binary ID, different revision) updated is attempted, it will raise an error. -The updated nodes will keep the "modified" flag when updated in *old_lockfile* - -This command is useful for distributed or concurrent builds of different packages -in the same dependency graph locked by the same lockfile. When one package is rebuilt -it will modify the package reference, and will be marked as "modified". The way -of integrating the information of package builds into the main lockfile is this command. - -Example: - -Integrate the information of building a "pkgb" package using a lockfile (and modified -in the folder pkgb_temp) in the main lockfile: - -.. code-block:: bash - - $ conan graph update-lock release/conan.lock pkgb_temp/release/conan.lock - - -.. _conan_graph_clean_modified: - -conan graph clean-modified --------------------------- - -.. code-block:: bash - - $ conan graph clean-modified [-h] lockfile - -Cleans all "modified" flags from the given lockfile. - -.. code-block:: text - - positional arguments: - lockfile lockfile folder - - optional arguments: - -h, --help show this help message and exit - -When a package of a dependency graph is going to be re-built, using a given lockfile, -it is desired to finish the build knowing which packages of the graph have been -actually rebuilt as a result of the last command. This command will clean all the -previously existing "modified" flags before such build, so after the build -the "modified" are only those that have been built now. - -.. _conan_graph_build_order: - -conan graph build-order ------------------------ - -.. code-block:: bash - - $ conan graph build-order [-h] [-b [BUILD]] [--json JSON] lockfile - - -Given a lockfile, compute which packages and in which order they should be built, -as mandated by the binary ID (``package_id()``) definitions and the ``--build`` argument, -which is the same as :command:`conan create|install` - -.. code-block:: text - - positional arguments: - lockfile lockfile folder - - optional arguments: - -h, --help show this help message and exit - -b [BUILD], --build [BUILD] - Optional, specify which packages to build from source. - Combining multiple '--build' options on one command - line is allowed. For dependencies, the optional - 'build_policy' attribute in their conanfile.py takes - precedence over the command line parameter. Possible - parameters: --build Force build for all packages, do - not use binary packages. --build=never Disallow build - for all packages, use binary packages or fail if a - binary package is not found. Cannot be combined with - other '--build' options. --build=missing Build - packages from source whose binary package is not - found. --build=outdated Build packages from source - whose binary package was not generated from the latest - recipe or is not found. --build=cascade Build packages - from source that have at least one dependency being - built from source. --build=[pattern] Build packages - from source whose package reference matches the - pattern. The pattern uses 'fnmatch' style wildcards. - Default behavior: If you omit the '--build' option, - the 'build_policy' attribute in conanfile.py will be - used if it exists, otherwise the behavior is like '-- - build=never'. - --json JSON generate output file in json format - - -The result is a list of lists, containing tuples. Each tuple contains 2 elements, the -first is a UUID of the node of the graph. It is unique and ensures a way to address -exactly one node, even if there are nodes with the same reference (it is possible for -example to have different build_requires with the same name and version, but different -configuration) - - -conan graph lock ----------------- - -.. code-block:: bash - - conan graph lock [-h] [-l LOCKFILE] [-b [BUILD]] [-r REMOTE] [-u] [-e ENV_HOST] - [-e:b ENV_BUILD] [-e:h ENV_HOST] [-o OPTIONS_HOST] - [-o:b OPTIONS_BUILD] [-o:h OPTIONS_HOST] [-pr PROFILE_HOST] - [-pr:b PROFILE_BUILD] [-pr:h PROFILE_HOST] [-s SETTINGS_HOST] - [-s:b SETTINGS_BUILD] [-s:h SETTINGS_HOST] - path_or_reference - -.. code-block:: text - - positional arguments: - path_or_reference Path to a folder containing a recipe (conanfile.py or - conanfile.txt) or to a recipe file. e.g., - ./my_project/conanfile.txt. It could also be a reference - - optional arguments: - -h, --help show this help message and exit - -l LOCKFILE, --lockfile LOCKFILE - Path to lockfile to be created. If not specified 'conan.lock' will - be created in current folder - -b [BUILD], --build [BUILD] - Packages to build from source - -r REMOTE, --remote REMOTE - Look in the specified remote server - -u, --update Will check the remote and in case a newer version - and/or revision of the dependencies exists there, it - will install those in the local cache. When using - version ranges, it will install the latest version - that satisfies the range. Also, if using revisions, it - will update to the latest revision for the resolved - version range. - -e ENV_HOST, --env ENV_HOST - Environment variables that will be set during the package build - (host machine). e.g.: -e CXX=/usr/bin/clang++ - -e:b ENV_BUILD, --env:build ENV_BUILD - Environment variables that will be set during the package build - (build machine). e.g.: -e CXX=/usr/bin/clang++ - -e:h ENV_HOST, --env:host ENV_HOST - Environment variables that will be set during the package build - (host machine). e.g.: -e CXX=/usr/bin/clang++ - -o OPTIONS_HOST, --options OPTIONS_HOST - Define options values (host machine), e.g.: -o Pkg:with_qt=true - -o:b OPTIONS_BUILD, --options:build OPTIONS_BUILD - Define options values (build machine), e.g.: -o Pkg:with_qt=true - -o:h OPTIONS_HOST, --options:host OPTIONS_HOST - Define options values (host machine), e.g.: -o Pkg:with_qt=true - -pr PROFILE_HOST, --profile PROFILE_HOST - Apply the specified profile to the host machine - -pr:b PROFILE_BUILD, --profile:build PROFILE_BUILD - Apply the specified profile to the build machine - -pr:h PROFILE_HOST, --profile:host PROFILE_HOST - Apply the specified profile to the host machine - -s SETTINGS_HOST, --settings SETTINGS_HOST - Settings to build the package, overwriting the defaults (host - machine). e.g.: -s compiler=gcc - -s:b SETTINGS_BUILD, --settings:build SETTINGS_BUILD - Settings to build the package, overwriting the defaults (build - machine). e.g.: -s compiler=gcc - -s:h SETTINGS_HOST, --settings:host SETTINGS_HOST - Settings to build the package, overwriting the defaults (host - machine). e.g.: -s compiler=gcc - - - -This command is similar to :command:`conan install` or :command:`conan info`, but -with a few differences: - -- It doesn't need to retrieve binaries, it will only compute what is necessary to do, according to the ``--build`` argument and rules -- Even when ``--build`` values are specified, packages will not be built from sources. It will just compute, as a "dry-run" what would happen in an equivalent :command:`conan install` diff --git a/reference/commands/misc/lock.rst b/reference/commands/misc/lock.rst new file mode 100644 index 00000000000..fd49ff1b98b --- /dev/null +++ b/reference/commands/misc/lock.rst @@ -0,0 +1,143 @@ +.. _conan_lock: + +conan lock +=========== + +.. code-block:: bash + + $ conan lock [-h] {update,build-order,clean-modified,create} ... + +Generates and manipulates lock files. + +.. code-block:: text + + positional arguments: + {update,build-order,clean-modified,create} + sub-command help + update Complete missing information in the first lockfile with information defined in the second lockfile. Both lockfiles must represent the same graph, + and have the same topology with the same identifiers, i.e. the second lockfile must be an evolution based on the first one + build-order Returns build-order + clean-modified Clean modified flags + create Create a lockfile from a conanfile or a reference + + optional arguments: + -h, --help show this help message and exit + + + +.. seealso:: + + read about lockfiles in :ref:`versioning_lockfiles` + + + +conan lock create +----------------- +.. code-block:: bash + + $ conan lock create [-h] [--name NAME] [--version VERSION] [--user USER] [--channel CHANNEL] [--reference REFERENCE] [-l LOCKFILE] [--base] + [--lockfile-out LOCKFILE_OUT] [-b [BUILD]] [-r REMOTE] [-u] [-e ENV_HOST] [-e:b ENV_BUILD] [-e:h ENV_HOST] [-o OPTIONS_HOST] [-o:b OPTIONS_BUILD] + [-o:h OPTIONS_HOST] [-pr PROFILE_HOST] [-pr:b PROFILE_BUILD] [-pr:h PROFILE_HOST] [-s SETTINGS_HOST] [-s:b SETTINGS_BUILD] [-s:h SETTINGS_HOST] + [path] + + +.. code-block:: text + + positional arguments: + path Path to a conanfile + + optional arguments: + -h, --help show this help message and exit + --name NAME Provide a package name if not specified in conanfile + --version VERSION Provide a package version if not specified in conanfile + --user USER Provide a user + --channel CHANNEL Provide a channel + --reference REFERENCE + Provide a package reference instead of a conanfile + -l LOCKFILE, --lockfile LOCKFILE + Path to lockfile to be used as a base + --base Lock only recipe versions and revisions + --lockfile-out LOCKFILE_OUT + Filename of the created lockfile + -b [BUILD], --build [BUILD] + Packages to build from source + -r REMOTE, --remote REMOTE + Look in the specified remote server + -u, --update Will check the remote and in case a newer version and/or revision of the dependencies exists there, it will install those in the local cache. When + using version ranges, it will install the latest version that satisfies the range. Also, if using revisions, it will update to the latest revision + for the resolved version range. + -e ENV_HOST, --env ENV_HOST + Environment variables that will be set during the package build (host machine). e.g.: -e CXX=/usr/bin/clang++ + -e:b ENV_BUILD, --env:build ENV_BUILD + Environment variables that will be set during the package build (build machine). e.g.: -e:b CXX=/usr/bin/clang++ + -e:h ENV_HOST, --env:host ENV_HOST + Environment variables that will be set during the package build (host machine). e.g.: -e:h CXX=/usr/bin/clang++ + -o OPTIONS_HOST, --options OPTIONS_HOST + Define options values (host machine), e.g.: -o Pkg:with_qt=true + -o:b OPTIONS_BUILD, --options:build OPTIONS_BUILD + Define options values (build machine), e.g.: -o:b Pkg:with_qt=true + -o:h OPTIONS_HOST, --options:host OPTIONS_HOST + Define options values (host machine), e.g.: -o:h Pkg:with_qt=true + -pr PROFILE_HOST, --profile PROFILE_HOST + Apply the specified profile to the host machine + -pr:b PROFILE_BUILD, --profile:build PROFILE_BUILD + Apply the specified profile to the build machine + -pr:h PROFILE_HOST, --profile:host PROFILE_HOST + Apply the specified profile to the host machine + -s SETTINGS_HOST, --settings SETTINGS_HOST + Settings to build the package, overwriting the defaults (host machine). e.g.: -s compiler=gcc + -s:b SETTINGS_BUILD, --settings:build SETTINGS_BUILD + Settings to build the package, overwriting the defaults (build machine). e.g.: -s:b compiler=gcc + -s:h SETTINGS_HOST, --settings:host SETTINGS_HOST + Settings to build the package, overwriting the defaults (host machine). e.g.: -s:h compiler=gcc + + +conan lock update +----------------- + +.. code-block:: bash + + $ conan lock update [-h] old_lockfile new_lockfile + +.. code-block:: text + + positional arguments: + old_lockfile Path to lockfile to be updated + new_lockfile Path to lockfile containing the new information that is going to be updated into the first lockfile + + optional arguments: + -h, --help show this help message and exit + + + +conan lock build-order +---------------------- + +.. code-block:: bash + + $ conan lock build-order [-h] [--json JSON] lockfile + +.. code-block:: text + + positional arguments: + lockfile lockfile file + + optional arguments: + -h, --help show this help message and exit + --json JSON generate output file in json format + + +conan lock clean-modified +------------------------- + +.. code-block:: bash + + $ conan lock clean-modified [-h] lockfile + +.. code-block:: text + + positional arguments: + lockfile Path to the lockfile + + optional arguments: + -h, --help show this help message and exit \ No newline at end of file diff --git a/reference/commands/output/info.rst b/reference/commands/output/info.rst index 98127949b1f..465c7312c59 100644 --- a/reference/commands/output/info.rst +++ b/reference/commands/output/info.rst @@ -21,7 +21,7 @@ Build order .. warning:: - The command ``conan info --build-order`` is deprecated in favor of :ref:`conan graph build-order`. + The command ``conan info --build-order`` is deprecated in favor of :ref:`conan lock build-order`. The build order printed with the argument :command:`--build-order` can be @@ -31,14 +31,14 @@ each nested one can be built in parallel. .. code-block:: json :caption: build_order.json - { - "groups":[ - [ + { + "groups":[ + [ "liba/0.1@lasote/stable", "libe/0.1@lasote/stable", "libf/0.1@lasote/stable" ], - [ + [ "libb/0.1@lasote/stable", "libc/0.1@lasote/stable" ] @@ -55,7 +55,7 @@ references. .. code-block:: json :caption: nodes_to_build.json - [ + [ "h0/0.1@lu/st", "h1a/0.1@lu/st", "h1c/0.1@lu/st", @@ -89,7 +89,7 @@ contain a list with the information for each of the nodes. "recipe":"No remote", "binary":"Missing", "creation_date":"2019-01-29 17:22:41", - "required_by":[ + "required_by":[ "libc/0.1@lasote/stable", "libb/0.1@lasote/stable" ] diff --git a/versioning/lockfiles.rst b/versioning/lockfiles.rst index 3c88db86ff7..9f9fa0c313e 100644 --- a/versioning/lockfiles.rst +++ b/versioning/lockfiles.rst @@ -1,287 +1,24 @@ .. _versioning_lockfiles: Lockfiles -========= +------------- .. warning:: This is an **experimental** feature subject to breaking changes in future releases. -Lockfiles are files that store the information of a dependency graph, including the -exact versions, revisions, options, and configuration of that dependency graph. As -they depend on the configuration, and the dependency graph can change with every -different configuration, there will be one lockfile for every configuration. -Lockfiles are useful for achieving deterministic builds, even if the dependency -definitions in conanfile recipes are not fully deterministic, for example when using +Lockfiles are files that store the information of a dependency graph, including the +exact versions, revisions, options, and configuration of that dependency graph. These +files allow for later achieving reproducible results, and installing or using the exact +same dependencies even when the requirements are not fully reproducible, for example when using version ranges or using package revisions. -Let's say we have 3 package recipes ``pkgc``, ``pkgb``, and ``pkga``, that define this dependency graph: - -.. image:: ../images/conan-graph_not_deterministic.png - -The first time, when a :command:`conan install .` is executed, the requirement defined -in ``pkgb`` is resolved to **pkga/1.0**, because that was the latest at that time that -satisfied the version range ``pkga/[*]``. After such install, the user can build and -run an application in the source code of ``pkgc``. But some time later, another colleague -tries to do exactly the same, and suddenly it is pulling a newer version of ``pkga`` that -was recently published, getting different results (maybe even not working). Builds with -version ranges are not reproducible by default. - -Using lockfiles ---------------- -Lockfiles solve this problem creating a file that stores this information. In the example -above, the first :command:`conan install .` will generate a *conan.lock* file that can be -used later: - -.. code-block:: bash - - $ cd PkgC - $ conan install . # generates conan.lock - # After PkgA/1.1 has been created - $ conan install . --lockfile # uses the existing conan.lock - -The second time that :command:`conan install . --lockfile` is called, with the lockfile argument -it will load the previously generated *conan.lock* file, that contains the information that -``pkga/1.0`` is used, and will apply it again to the dependency resolution, resolving exactly -the same dependency graph: - -.. image:: ../images/conan-graph_locked.png - -The *conan.lock* file contains more information than the versions of the dependencies, it contains: - -- The "effective" profile that has been used to compute this lockfile. The effective profile is the - combination of the profile files that could have been passed in the command line, plus any - other settings or options directly defined in the command line. -- It encodes a graph, not just a list of versions, as different nodes in the graph might be using - different versions too. -- The options values at each package. As downstream consumers can define options values, it is - important that this information is also stored, so it is also possible to build intermediate nodes - of the graph leading to the same result. -- Another kind of requirements like python_requires. - -Again, it is important to remember that every different configuration will generate a different -graph, and then a different *conan.lock* as result. So the example above would be more like the -following if we wanted to work with different configurations (e.g. Debug/Release): - -.. code-block:: bash - - $ cd PkgC - $ cd release - $ conan install .. # generates conan.lock (release) in this folder - $ cd ../debug - $ conan install .. -s build_type=Debug # generates conan.lock (debug) - # After PkgA/1.1 has been created - $ conan install .. --lockfile # uses the existing conan.lock (debug) - $ cd ../release - $ conan install .. --lockfile # uses the existing conan.lock (release) - -Commands --------- - -There are 2 main entry points for lockfile information in conan commands: - -- :command:`--lockfile` argument in :command:`install/create/export/info` - - If the command builds a package, it can modify its reference. Even if the version is not changed, - if something in the recipe changes, it will get a new recipe revision RREV and if the package is - built from sources again, it might end with a new, different package revision PREV. Those changes - will be updated in the *conan.lock* lockfile, and the package will be marked as "modified". - -- :command:`conan graph` command - - This command group contains several functions related to the management of lockfiles: - - - :command:`conan graph lock` - - This command will generate a *conan.lock* file. It behaves like :command:`conan install` command, - (this will also generate a lockfile by default), but without needing to actually install the - binaries, so it will be faster. In that regard, it is equal to :command:`conan info` that can also - generate a lockfile, but the problem with :command:`conan info -if=.` is that it does not allow to - specify a profile or settings. - - - :command:`conan graph clean-modified` - - When a :command:`conan create` command that uses a lockfile builds a new binary, its reference - will change. This change, typically in the form of a recipe revision and/or package revision - is updated in the lockfile and the node is marked as "modified". This :command:`clean-modified` - removes these "modified" flags from a lockfile. This operation is typically needed before starting - the build of a package in a locked graph, to know exactly which nodes have been modified by this - operation. - - - :command:`conan graph update-lock` - - Update the current lockfile with the information of the second lockfile. Only the nodes marked - as "modified" will be updated. Trying to update to the current lockfile one node that has already - been "modified" will result in an error. - - - :command:`conan graph build-order` - - Takes a lockfile as an argument, and return a list of lists indicating the order in which packages - in the graph have to be built. It only returns those packages that really need to be built, - following the :command:`--build` arguments and the ``package_id()`` rules. - -For more information see :ref:`commands` - -How to use lockfiles in CI --------------------------- - -.. note:: - - The code used in this section, including a *build.py* script to reproduce it, is in the - examples repository: https://github.com/conan-io/examples - - .. code:: bash - - $ git clone https://github.com/conan-io/examples.git - $ cd features/lockfiles/ci - $ python build.py - -One of the applications of lockfiles is to be able to propagate changes in one package -belonging to a dependency graph downstream its affected consumers. - -Lets say that we have the following project in which packages ``pkga``, ``pkgb``, ``pkgc``, ``pkgz`` and ``app`` -have already been created and only one version of each, the version 0.1 exists. All packages -are using version ranges with a range like ``pkgz/[>0.0]``, so basically they will resolve to -any new version of their dependencies that it is published. - -Also, the ``full_version_mode`` will be defined for dependencies. This means that if the version -number of one package dependencies change, then it will require a new binary. This assumption -is reasonable, as PkgA, PkgZ are header only libraries and PkgB and PkgC are static libraries -that inline functionality defined in PkgA and PkgZ. No matter what the changes in PkgA and PkgZ -are in new versions, it will be necessary to build new binaries for the downstream consumers. - -.. code:: bash - - $ conan config set general.default_package_id_mode=full_version_mode - -Now, some developer does some changes to PkgA, and do a pull request to the develop branch, -and we want our CI to build the new binaries for the dependants packages, down to the final -application App, to make sure that every works as expected. - -The process starts generating a *conan.lock* lockfile in the *release* subfolder: - -.. code-block:: bash - - $ conan graph lock app/0.1@user/testing --lockfile=release - -This lockfile will contain the resolved dependencies in the graph, as we only have one version -0.1 for all the packages, all of them will be locked to that 0.1 version. - - -.. image:: ../images/conan-lockfile_ci_1.png - - -Once the lockfile has been generated, it doesn't matter if new, unrelated versions of other -packages, like **pkgz/0.2** is created with ``cd pkgz && conan create . pkgz/0.2@user/testing`` - -Now we can safely create the new version of **pkga/0.2**, that will resolve to use **pkgz/0.1** -instead of the latest 0.2, if we use the lockfile: - -.. code-block:: bash - - cd pkga && conan create . pkga/0.2@user/testing --lockfile=../release - # lockfile in release/conan.lock is modified to contain pkga/0.2 - -Note that the lockfile is modified, to contain the new **pkga/0.2** version. - -The next step is to know which dependents need to be built because they are affected by the new -**pkga/0.2** version: - -.. code-block:: bash - - $ conan graph build-order ./release --json=bo.json --build=missing - [[PkgC, PkgD], [App]] # simplified format - -This command will return a list of lists, in order, of those packages to be built. It will be -stored in a *bo.json* json file too. Note that the ``--build=missing`` follows the same rules -as :command:`create` and :command:`install` commands. The result of evaluating the graph with -the **pkga/0.2** version, due to the ``full_version_mode`` policy is that new binaries for -PkgB, PkgC and App are necessary, and they do not exist yet. If we don't provide the ``--build=missing`` -it will return an empty list (but it will fail later, because binary packages are not available). - -We can now proceed iteratively with the following procedure: - - -1. pop the first element of the first sublist of the build order result, get its ``ref`` reference - - .. code:: python - - # python - _, ref = build_order[0][0] - ref = ref.split("#", 1)[0] - -2. allocate some resource, like a CI build server, or create a temporary folder. - - .. code:: bash - - $ mkdir build_server_folder && mkdir build_server_folder/release - -3. copy the lockfile to that resource (and move to it) - - .. code:: bash - - $ cp release/conan.lock build_server_folder/release - $ cd build_server_folder - -4. build the package - - .. code:: bash - - $ conan install --build= --lockfile=release - -5. go back to the parent, update the lockfile with the changes - - .. code:: bash - - $ cd .. - $ conan graph update-lock release build_server_folder/release - $ rm -rf build_server_folder - -6. compute again the build-order of packages, if not empty, goto 1 - - .. code-block:: bash - - $ conan graph build-order ./release --json=bo.json --build=missing - -7. clean "modified" nodes from the lockfile - - .. code:: bash - - $ conan graph clean-modified release/ - - -Note that this is a suboptimal approach, in order to explain the functionality, which -is more easy to follow if it is sequential. In reality, the CI can take the first -sublist output of :command:`conan graph build-order` and fire all its packages in parallel, -because they are guaranteed to be independent. Then, as soon as they start finishing and -build servers become available, the :command:`conan graph build-order` can be reevaluated, -and new builds can be launched accordingly, just taking care of not re-launching the same -build again. Note that the result of build-order contains a unique UUID, which is the identifier -of the node in the graph, which could be useful to dissambiguate. - -.. image:: ../images/conan-lockfile_ci_2.png - -With this later approach, a deterministic build with optimal Continuous Integration process -with optimal utilization of resources and minimizing unnecessary rebuilds is achieved. - -Note that this example has been using incremental versions and version ranges. -With package revisions it is also possible to achieve the same flow without bumping the versions and using fixed version dependencies: - -- It will not be necessary to change the recipes or even to inject the values in CI. - Every change in a recipe will produce a new different recipe revision. -- Revisions are also locked in lockfiles. -- As revisions are resolved by default to latest, and the conan cache can only hold - one revision, it might be necessary to pass ``--update`` argument so the correct revision is updated in the cache. -- It is necessary to define the ``recipe_revision_mode`` or the ``package_revision_mode`` if we want to guarantee that the binaries correctly model the dependencies changes. - -For implementing this flow, it might be necessary to share the different ``conan.lock`` lockfiles among different machines, to pass them to build servers. A git repo could be used, but also an Artifactory generic repository could be very convenient for this purpose. -.. note:: +.. toctree:: + :maxdepth: 2 - There is a **very experimental, temporary** configuration (``general.relax_lockfile``), that allows to expand dependency - graphs with packages that are not in the lockfile. This scenario happens for example when a ``test_package/conanfile.py`` contains - other requirements. If the lockfile was built from another downstream consumer, the ``test_package`` and its requirements will - not be contained in the lockfile. But we might still want to do a ``conan create`` for that node of the graph. - Putting the ``general.relax_lockfile=1`` will allow this case. This is a temporary thing introduced at 1.23, will be removed in future versions - (while probably leaving the behavior in some of its forms) \ No newline at end of file + lockfiles/introduction + lockfiles/configurations + lockfiles/build_order + lockfiles/ci \ No newline at end of file diff --git a/versioning/lockfiles/build_order.rst b/versioning/lockfiles/build_order.rst new file mode 100644 index 00000000000..dc0cba00294 --- /dev/null +++ b/versioning/lockfiles/build_order.rst @@ -0,0 +1,216 @@ +.. _versioning_lockfiles_build_order: + +Build order in lockfiles +======================== + +.. warning:: + + This is an **experimental** feature subject to breaking changes in future releases. + +In this section we are going to use the following packages, defining this dependency graph. + +.. image:: conan_lock_build_order.png + :height: 200 px + :width: 400 px + :align: center + + +.. note:: + + The code used in this section, including a *build.py* script to reproduce it, is in the + examples repository: https://github.com/conan-io/examples. You can go step by step + reproducing this example while reading the below documentation. + + .. code:: bash + + $ git clone https://github.com/conan-io/examples.git + $ cd features/lockfiles/build_order + # $ python build.py only to run the full example, but better go step by step + + +The example in this section uses ``full_version_mode``, that is, if a package changes any part of its version, its consumers will +need to build a new binary because a new ``package_id`` will be computed. This example will use version ranges, and +it is not necessary to have revisions enabled. It also do not require a server, everything can be reproduced locally. + + +.. code-block:: bash + + $ conan config set general.default_package_id_mode=full_version_mode + +Let's start by creating the initial dependency graph, without binaries (just the exported recipes), in our local cache: + + +.. code-block:: bash + + $ conan export liba liba/0.1@user/testing + $ conan export libb libb/0.1@user/testing + $ conan export libc libc/0.1@user/testing + $ conan export libd libd/0.1@user/testing + $ conan export app1 app1/0.1@user/testing + $ conan export app2 app2/0.1@user/testing + +Now we will create a lockfile that captures the dependency graph for ``app1/0.1@user/testing``. +In the same way we created lockfiles for a local *conanfile.py* in a user folder, we can also +create a lockfile for a recipe in the Conan cache, with the :command:`--reference` argument: + +.. code-block:: bash + + $ conan lock create --reference=app1/0.1@user/testing --lockfile-out=app1.lock + +The resulting *app1.lock* lockfile will not be able to completely lock the binaries because such +binaries do not exist at all. This can be checked in the *app1.lock* file, the packages do not +contain a package revision (``prev``) field at all: + +.. code-block:: text + + { + ... + "4": { + "ref": "liba/0.1@user/testing", + "options": "", + "package_id": "5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9", + "context": "host" + } + ... + } + +We can now compute the "build-order" of the dependency graph. The "build-order" lists +in order all the packages that needs to be built from sources. The logic is the following: + +- If a package is fully locked (it contains a package revision field ``prev`` in the lockfile), + it will not be built from sources and will **never** appear in the build-order list. +- If a package is not fully locked (it does **not** contain a package revision ``prev`` in the lockfile), + it will appear in the build-order list. This situation happens both when the package binary doesn't exist yet, + or when the ``--build`` argument was used while creating the lockfile. + +.. code-block:: bash + + $ conan lock build-order app1.lock --json=build_order.json + +The resulting *build_order.json* file is a list of lists, structured by levels of possible parallel builds: + +.. code-block:: text + + [ + # First level liba + [["liba/0.1@user/testing", "5ab8...1ac9", "host", "4"]], + # Second level libb and libc + [["libb/0.1@user/testing", "cfd1...ec23", "host", "3"], + ["libc/0.1@user/testing", "cfd1...ec23", "host", "5"]], + # Third level libd + [["libd/0.1@user/testing", "d075...5b9d", "host", "2"]], + # Fourth level libd + [["app1/0.1@user/testing", "3bf2...5188", "host", "1"]] + ] + +Every item in the outer list is a "level" in the graph, a set of packages that needs to be built, and +are independent of every other package in the level, so they can be built in parallel. Levels in the +build order must be respected, as the second level cannot be built until all the packages in the first level are built +and so on. In this example, once the build of ``liba/0.1@user/testing`` finishes, as it is the only +item in the first level, the second level can start, and it can build both ``libb/0.1@user/testing`` and ``libc/0.1@user/testing`` in parallel. It is necessary +that both of them finish their build to be able to continue to the third level, that contains +``libd/0.1@user/testing``, because this package depends on them. + +Every item in each level has 4 elements: ``[ref, package_id, context, id]``. At the moment the only +necessary one is the first one. The ``ref`` value is the one that can be used for example in a :command:`conan install` +command like: + +.. code-block:: bash + + $ conan install --build= --lockfile=mylock.lock + + +Defining builds +--------------- + +The definition of what needs to be built comes from the existing binaries plus the :command:`--build` +argument in the :command:`conan lock create`. + +Let's build all the binaries for the exported packages first: + +.. code-block:: bash + + # Build app1 and dependencies + $ conan install app1/0.1@user/testing --build=missing + + +Now that there are binaries for all packages in the cache, let's capture the in a new lockfile and compute the build order: + +.. code-block:: bash + + # Create a new lockfile now with all the package binaries + $ conan lock create --reference=app1/0.1@user/testing --lockfile-out=app1.lock + # And check which one needs to be built + $ conan lock build-order app1.lock --json=build_order.json + # The build order is emtpy, nothing to build + [] + +The result of this build order is empty. As the :command:`conan lock create` found existing binaries, +everything is fully locked, nothing needs to be built. + +If we specify the :command:`--build` flag, then the behavior is different: + +.. code-block:: bash + + $ conan lock create --reference=app1/0.1@user/testing --lockfile-out=app1.lock --build + # the lockfile will not lock the binaries + # And check which one needs to be built + $ conan lock build-order app1.lock --json=build_order.json + [[["liba/0.1@user/testing", "5ab8...1ac9", "host", "4"]], ... + + +This feature is powerful when combined with ``package_id_modes``, because it can +automatically define the minimum set of packages that needs to be built for any +change in the dependency graph. + +Let's say that a new version ``libb/1.1@user/testing`` is created. But if we +check the ``libd`` *conanfile.py* requirement ``libb/[>0.0 <1.0]@user/testing``, +we can see that this 1.1 version falls outside of the valid version range. +Then, it does not affect ``libd`` or ``app1`` and nothing needs to be built: + +.. code-block:: bash + + $ conan create libb libb/1.1@user/testing + $ conan lock create --reference=app1/0.1@user/testing --lockfile-out=app1.lock + $ conan lock build-order app1.lock --json=build_order.json + [] # Empty, nothing to build, libb/1.1 does not become part of app1 + + +If on the contrary, a new ``libb/0.2@user/testing`` is created, and we capture a +new lockfile, it will contain such new version. Other packages, like ``liba`` and +``libc`` are not affected by this new version, and will be fully locked in the lockfile, +but the dependents of ``libb`` now won't be locked and it will be necessary to build them: + +.. code-block:: bash + + $ conan create libb libb/0.2@user/testing + $ conan lock create --reference=app1/0.1@user/testing --lockfile-out=app1.lock + $ conan lock build-order app1.lock --json=build_order.json + [[['libd/0.1@user/testing', '97e9...b7f4', 'host', '2']], + [['app1/0.1@user/testing', '2bf1...e405', 'host', '1']]] + +So in this case the *app1.lock* is doing these things: + +- Fully locking the non-affected packages (``liba/0.1``, ``libc/0.1``) +- Fully locking the ``libb/0.2``, as the binary that was just created is valid for our ``app1`` + (Note that this might not always be true, and ``app1`` build could require a different ``libb/0.2`` + binary). +- Partial locking (the version and package-id) of the affected packages that need to be + built (``libd/0.1`` and ``app1/0.1``). +- Retrieving via ``build-order`` the right order in which the affected packages need to be built. + +Recall that a package in a lockfile is fully locked if it contains a ``prev`` (package revision) field defined. +Fully locked packages cannot be built from sources. Partially locked packages do not contain a ``prev`` +defined. They lock the reference and the package-id, and they can be built from sources. + + +If we want to check if the new ``libb/0.2`` version affects to the ``app2`` and something needs to +be rebuild, the process is identical: + +.. code-block:: bash + + $ conan lock create --reference=app2/0.1@user/testing --lockfile-out=app2.lock + $ conan lock build-order app2.lock --json=build_order2.json + [] + +As expected, nothing to build, as ``app2`` does not depend on ``libb`` at all. diff --git a/versioning/lockfiles/ci.rst b/versioning/lockfiles/ci.rst new file mode 100644 index 00000000000..9a954e47715 --- /dev/null +++ b/versioning/lockfiles/ci.rst @@ -0,0 +1,10 @@ +.. _versioning_lockfiles_ci: + +CI +======================= + +.. warning:: + + This is an **experimental** feature subject to breaking changes in future releases. + +TODO \ No newline at end of file diff --git a/versioning/lockfiles/conan_lock_build_order.png b/versioning/lockfiles/conan_lock_build_order.png new file mode 100644 index 0000000000000000000000000000000000000000..5c1c2ea46a9e8f9db6abc4fc53ac120ac4edfee6 GIT binary patch literal 13121 zcmeHuX*`r+xc5lWCN0`Vp~zB}VkFrUveQ1+vNqNtWG5w*eG6qDRF)cB)@f%fVQ9oy zGn8e>*bU}fqdM>T`hIxd59gc@{pQ#6+}m?)_x1l@mrw&e4UVnItq25yL-WG@SlygD!M8N#GA-%D>pa6|F_(}fObV7c6?<1 zW2tk_vqB&wNt)+XF5few_M)TqUa6hB(0mA7Uc7*U9&m4Htz_&T%K7Yvdt;fUm%)QLI=N1cGu71E;GnKq3&z;&Jdb zQQNRA?}ItG)#KO$cEqu5M8CcfINI?dcBy=uxPTUazyFdf!oU|^Ok;2M&hwwYDJWf| zi9v8>p!hapRXl__0$`bq8K_@7uq>?)i2;0ZY`qyMM5O_8=L2H8TpXM7el6Df2FSaQ zh=^b3Fv>f%f+k^iQ(=zH=P<2@wOG^P04l;9m8uxT-T!8b-gcVgmg+LO-PYgyfrBh5 zQFQUIxI*8cs&O|8mZ(#VTdAqCb$&HIjXbWHM4r<$kAo-1-c3%unhH3Qlz2_-KjrsT zZ@Kmu&0|T4rl}Y~IvT0ZW1HV%@>h}-wEs$O&L(JGX)2<7!aZoNJ?0;&jXIMT<7#+m+`;M7kCn>*5G8%8k#7il8>}WyS$Xd7P_L~si2e~%HgL|I9?~< z!l3!b`W%YMC*KkQdw!D+^OLkyNyP{P6}Mv<&F)hnKVD`RACyma{4WE7Zb6@5=G zlnVKCedk-|tv+Tz`JGnf=w~ z?n9}Mo*%FEr6<3!tm}4ET%En-eGqYZ6HNI0C7iptN06n&S!PZye2rFKoy8YN^FdR!whc6O5Eqyj08NS;zMtMWlZWI$XHJ7@C4 zZTpi?S$=tXqVVkpo|Ra&ceax~6_ZG%?$+SWnohY>Z&RLOR!`~I`EL6nYcrw4o@^FVeKnIt2m=Vse1q@Vc}uW- z^=`tA)qBS)3nG!Wuxq$aF(Vx<^M`XLBtGlP$KlT(~32|OLc&pAE@ zbI6Ua1k1XPycL__mT`LKn{>`B;qA;Hdp@ zj~oqI0t~>t8;mePu;Y-RP_KbSm+>9)?urH$WO3bgr38PzdR} zQ+Igeu|)0dtZ!eK%Y_mHi1c}~GnHLc^JdiN=#rbT&Kz9vFC2T`w#`oU_pjv?tj_r+ zWU?8cv{<#qzg;sg#hdnf^!tvyDyntKs(iyG9dOhTX>gXElLy7G{^T|Lm(ljzDno(w z7qIhj+-t)$MPG-j1xCZ_hL6p|mSI0z*yg=3hcJidRh1n9G|?b=a)3{awxRy>v+>wG zmJ)6(OIz&BUP|>Q*Ptk_ewap^+7Kmu;LKY$Q8;uf_|tiftFF~xyN)-buJ9OcXFk0OPt_D9GPM)M=c)H; zvb%+Ov<+G7K?TC~LWYXdpZ3_A*(TR1v)i~0R#j&9}Cc28L&gH6)c1V-U99^?+2`~i){>gD?;`=En_lfJ?H@qe*e)b4k ze1PSM!W?;39Ned$`mHV}r7CzQTQ??D7=8+aFTWSZ*72<=Ns7wyYmGi7Q{^$`o)vN! z4!09tJj`d|vmI_!$$jyH5T5B^=1rBRD;zp`0|D>1f)38e@|%-b-a_P&ZN04QA~HsZ z6)H9oO5coSdFVCxaR0=6>yA9bk=6`VC$1||CSl7p3w^Fxh?h@@0p}*VOB-uV^|uDe zW5eyVB-DHd-m!RW2{@jKO8+Q^#(i32JI?#Qr>AE^-Ow67`xn^SBd_JrT&nZ)p}I(! zx=3D+tkwW*Uf@_3DqYoQZRw{?ebkZpfN9WnmEkQgUS&5!!q}Ok8kX`D|kMrI)?$HQBy~4lufDB(kHtOD0f{CqI8cES*2G{3|_uSG2^n=T+#+EacmSp+dBFJr7@ecgdLy?nM{ku! zd73t;O?d^>hJn~@tN8R6Hqp!_OSw)BQoo}gYsGfVNHHsQ>y-slv9?$+?!@0B&81- zB9Bv67y4ytyyhfjdTs}y@qRV%&H~<9!n=*tSpfBb-eZn>6H|bH{9Z8 zJ}tb9y$Nx6UdEvyy8lV#-BAni3d(LN7fOAT<4Yc{tcnWR8Aeq@lB{daLDa1%F;1M( z%Y5;q(&q_?0ufWpVn5?2BU$hH*A{e{XnQrZ%c>`n=J2o)xcwF$0b+8s7FU*mBIRnM@XdT5-RQ`@# zhj)CCjfq?l=J@;v_jwv1FT6O15#F&52|8B_!f)(~maPG1crxHKT9zH1mDeby^KI*rXnQIDjfH-$PeaKywv`u`u zs=~R+ie8%LAOJWqw$X~a>$GR@U90FQaGa$P}M$#t0({;XU1G$Mq?>!#8Z>qdpnYXvFNd?Ra6R@f|URgI}@E^>bOvHH5M zHh^-RTBlJ`|H}>8Q5RuN2fB)c$K)^07}%5~L4e z%NvuwQzW|tWlw!e_Oa8oC>5+R0{6bWe!@%kygE6*R$r2+b$<_-LlQ`Mc8DUGWD*m5 zmGpqWGxiN-=falC<1(4(v2$Hrg6^$ zVt~*8hQ<#BZz3~Bt~h@%7_g94gDI4`9#@X>l!jS zC8hcPrfrvPAwy+?-;u(qNlAT$xws@LTfGtYE6sx+ANA_9o+XZbdnLudOApu5C#9HI zIV<}PChj;180^$k^^D#~*_G|B$;5cJm0JL}++D_dDvFySb#U(Unb9qF{i#>JDPaD9 z{Jqzs-KDQ8CUA{kC(MJ`IB=VM2{Fe;#*6Uo#v7ImkXmEy`br8?8JaPBOn_LRI$z-R z`5j0`Dkm^`f>Qg=WH?xIG`~hN&&5y;3j@EcDoVNV(907!-7zMm+n zb|GaaJV_g<^q6wu@ucjccMJuxviC@{_!=GGJZig-2U1@t|Dt$t)5`BkDapw_3W#qS z7~u-Jsp@vUd!t*#o-g!yx0tfFYJ6o()6Z4%3GgRsX6(F_tJmD~MMLW&Bw!BZn@ZAA zaXmt5J~6Tx$7Xt5TPg}zo}YH?j;nAM;D4B5j`th+{>F0B=u409La%L&X!KX0O?nf6 zFAm-sm1|rqh)UY%j9+yx->b?_7KZ05GZYnmLkj^FqJ>RN2P6 z>8Q(}4tw-OFG+p|2w}hD@L5XPhFK=Cv)_jxWU4*xNwD>1tS+>4m0c?`n?D!=V(k+( zukCT%a+w#KaTI5k!%Xt2=)O)f?*eqOO*naXt1XZROs?nW1izw0_3!&+$rO{+9=~rd zugn)D;%vfv6mEXlI0GO1wjgjU6LmSnKj~cRecvCULKEMCiD-GYl24sOn#;+8e zh#2l-t1uNgGGUo_Tub$HeW*!_jI+D>b4OrO<|l5W?D~orkC?yk?w5GoO=01bsEDez6ujA2d!L*xi9Lc9; zw#LPZCMjCg@1#qKITx%j-9ec?ptct8>^89oiZ+UF8PVML(;lS$z zjjN4IvI<_aul*yF;x@@@qUVcAeSKR#Js~3OfsB~{bP_%6)R-XV#1#^GD8Sm&Ru<7n z=4~udyw8Y|^$d1iU7iCx1cB*m)Sfbm$mOJtGc(+8qGz}$KAV=7;)dHAm)oXrX@o5? zw5fZ5JON2y&tcxlruxpKE-ln&UJ_Lc z#{sY0PQ+?c?LXrdInpSP)T5>MSru#&c{B7Syx9KaWlcaKbv)jHcy=jhvD(GxFUDeB zt_4J9_hr7E1NOQYrIOUvXgn3CGnp0|C+9JFH}Mw$ zr;vj%>ahI1Rz)&jY^6LMaW^i}+^=iRgdmFo$+>2E0g$l(ESs{lkleVm!=;b4)r^+^ z?Oh;i3s)PQqDq?&fZ-cBlPTr!UYhY>{53O7KZd2`GtdkZ#$LlW76*bjlL3YHnBEr! znLs(9)d?3IRL!9$D1|1o)LWAB(K*121O1)v*al^H2Ddlj3Bd8xRBF_yI608BCr?E|8!V$h5Pp5JK4A z6HNgutP=~PpZA;J%@PK3K;7FA*?OC1=C?$h3p-JG^6`EN_pDneH0l0uDPSO-Hq&#gaze3y`_(wN^i9dfJ_Rn7Ul?v5Z!l|>{#Bl9jNZlTWk$^SAa9V(5DKdQ z!m5e))VWS`kE~k~OHb_>q7gp3U-&$cj(UjAiw157v)CdV6~B4--J`M;Igdi$8l%3x z2It87$Jo31lA6wIZh`l!ff-|tP;EOcv{f88Z}?{|9D#(Lm!Kpdk$6V`3# zRqW9-lfbXwxYhQdU)wgsiOe&!`l3IK`EUMwv7-_eGm@y8jd}i-$LF_7vB#8|%vSw< z)-jxHv1-Xy6;A2=Z@5bJ;@IMntIr>KwXcD^3Ee!Y;et!{nSHDC%d#$FSj<&;ySCxD zDr|UI(dT<@nu%cH&ouwO@G${i;bRO=8p|c0i8Etrq~#kTc89@aw|mOv96C8nO_Fn; zxSe&jeoN*meW9G;=h?-US6)=R-0`=wJOdHjNPy_ZW82O#NvZIwaBEjh z7#mn;@|?1GVA{UnmH;}el6mVNQ%%S$Hmkh*f_oDZ zWb%qhdV4|XgPJXU@Ar+F(cHAkHkvWy0~Nyw#uIqmuY+eiY1vg9L$qChCfjHn$BSi& z7qlLPOrxCryGNgAU3fw}RE3%rr-!Kc;Xrntyvw`JxnUDC`^p?ssy14m2X@|%7?Z1G zz57H8)IUC^=C)ONI7|7j`M|vh)ZjHJsoT|uM$>k+(%jE4J*boX{`OARK}oBS=eN>4 zM-I+V%lj*}16(Xb#$2*c%JIzV)H+3H1O&cI)^v2K5ODKWyt&m6<03a{#J}cp^Mu+& z48u|5>qG=~mEy7ivRMUY)kDjlhmA=$iSuf)b{M^}c<3up;iLGmyVSn-){sMA^-@zY zpT7L`179-Iz)e)m6pHeBf>RfMr?t0sbWqgCwe^~qZbY?=^6+x_{`$S?hF|Y5b_^?H z5NnN34oXd^822>@^VXP3;4I95ai6dfX}rOj{D#YI1u8uJMK2nhWpNuPan_AJw@y(+ zx*(w@6Yli9z2hc!TzP`x_*qo#FvQNN+q1t6rEHoGc%Mv@v<~B@p2^jFjwxEcCS;rQ?!&`75e1eaV{Z+R z0Z6OCgPy0A1y3wdYUg7jM|q^};>5Pma`X4H@F^u^MzBL2Z@oD4DRd9jN*W4muFJC+ z4^73jjoO~U8NX%(0$##WU$RgL*{0`moqCs{Linny((M&@fg@yywt_OUk1`6ZS(P0{ z)q2mssf0Z1f*6NFjEV{xs`x0qJ8|LFa_0(J(}zGFmT;oeF`N-Gp;`^^p_K7cJ$zMk z8`?Po?!0Q zwe_sqShl1RW2vRNe{Ei7CenZ9qlpXQ*hnIN#Tt-HOOfB4gvyk-N?qlTuoIGaeI%`_ zfA#CiOd12Vj*5njXhr_Zypy>ZKy&W=lo+h3>-@5L@u!Vdgng(FLxf}c?c$He&1npm zn{YSe^|ov;Y_kHc<~Q|m}kiB`q;_4PWA1SUjO;= zhX%}4XR?8+0H&eu z*3{Sttq+;|EP=pcbJ>|#C+y$QDJh=V=uQ|P*6n9eZNi#*JsmclFBzvDf5;TK>Ugy8 zrFp}yYsKOcuIt(xoM^;3k@XE|2Ygm6341EUD)hX0oue~QpEN8nzv&_RE{v8i`E9~2 zY=R4_O8Sc>*hTtkZ7ID#1Ew1t)KWc*W}h$1#uEAV=+m%U%n!tn>4X=h^o7WFeG4F5 zG73=jYjHhz+M##tt`GW!hV_@ar^>~8gClzq`ZZ=8% zwi(2`IT0Y{wp{nGwEZY$hCH3M zpkJ_tG2APDf)M0spZM)K``TK&b>>ydsS!^HwMTH@$5){0hvxU(XEZB%*34!PmPe0& zJHCN1PNxbcR__lzbX|H4%kd=1^)o@}i)S$tI_k*qa|r@h6M z%`L?Dlwq?M80ySS0#?oz3Yv9C$0`?ueu@;@b{`MUKPFO^C`44kdoJLs$yngm;;pFK zG!HFRzdsEp7xWIHherD^_~5=23rhNX$FX%>%E!K+V*fLNe+Yd{`6%6Z8D~Vkgv5dG zXNO@`1j`cypX3vxq(QhXkxNiPjZ;r#!%x+>s7ZQis~VSUf0?>gIv;OD7snof%JTi+ zIv@38wY_%RV~8)L4Xv@Hq3|Q%zbG7PG3^_rrJ8q!uF!0`It}t_{ zN)UXUzeTf5UtitDlk`Avx*W109#{*stMrQC`(=t1L{c`HOL&uC@7U3;&g1ODm|1x4 zrnL4ZyPYU+&;ngcrILwkOGbhS&y|?`EuYOho%uVX$qQsH%2cV~ZcP_jtbmXZZ-t*r zPk$j^Px6$Gjv`YE%g@-lG-8yBRD&ZG?0Ki@K@%EcRDaKMZL@@k{?neuYhI)E*3c?d zH_oK286+#C$|5jke8gC4qv>!}#Z@ zcnfn(ht{B?EdB?_&;O^_P_hIEW9I(0ZzDhO3=Oz5Av7Pw9vP>fpJl250IZH%VeRr* z(;ZzEc_1LA@EYg!0hjgYtJXoL!QxKswm$1I=@4;9_J=wmy%LN&09e0n;_t6=Q*U=6 z^2D`RcMbsB!eIv^WV2o2 zFdzD=5CAG(V+V&KmE8Zxhwt6j=W>pi@67TmO8_u=e*~@lQG?f)@>k3V&%_6D!JJAf zt^Zar_g^g)<7jSfcG`$|`Iv~<_#K!!F%IZXy8CZgpg0^bOn#$hGvYF|f^2>b;K>+y z+Tr`aKkfO_=evln{Z_qXHXy#K!Eq=0@%;(GZsVkif9eD-r3{sCT1EP|Gc>l8U~e*8 zPyX)wn>QZV3A=ww_y8vfGmJ_b{>^yA$yQW=RRbOd|N-F8#g>&&4=^s&V_b%=A#5 zBj!5JD)gURnJocYtO#kQ;qv+|zE24L{CI!Wzb}Pcy9_1D1aW^gC8VDUUpn~V;eVbR zh*2>PD4PE}_dDqAal|}jGK7Cr7nqG|N|9;t-SkJCrqFz6!)%#5vF;rASI5)VteE-n z=|mHkP-e|rN}@d`Cr5Ch9g*I^B$U|0`&QZaV56rPfFOGGSp$wkchA#9C}>f}te3%q zq30NGi<5tPjul|edfv&J4wwO2h?qUcvq0<}_|wP?fMnx(Zp+ddL=3pGp4cM1;P!a( zr;!=x#rx|i?SH#PGv-8lUN^5dD*IXr+M1b-KyT_FaotjYda|V-ovxI8ZC`$Kb#)U2 z4yI6a>@iUr_|yi7;tEYPF>lH8=~1~G^|EBb)0O_El3{d6S&v8Czp;?K^87ZIyKP&8 zI?|bVszRrB{+RNY)zhWK1bU$c*EV{Hw~CL2)hKA`6$Vcw{{m*y|% zpHr^AFEqbou6BGEGh8Ro2#ldaLBzKo2huFD%<$@or z-GStO58URC%833lY)s5{=%=oML@PfUd*sAuKV|j(^&L0@m7<&~y15pW?ceS^I?&S{ zD7ShY%H*lgOj7Ld+v3SYSfuHXp{vKU$-Zk?72<3f4e}wqm$R?N5?fPkk#?r~ zg|X7&k^HO97-oC(9MDa#)#HPW@THx{IV2~`3EkqWrP(Ck#g4PL>Enc9!X3>57lT(i z9b5l>jv-bhMzM97)#WNC>W=v#EAI+3TDo{5wus#A%qZQ_v!+;F_;Z^Y&1=ms=JOY& z)UrO08ve*=a4n!fPs-jxy+kdtzLsYTSE^zGLh|m80yKirPcjzJ#c3 z&hE7vseADS_yVIAWHzq{tx^$t b%HT9T^4;{q%xgT%Lujh$oi99h ... pkga/0.2@user/testing from local cache - Cache + # Example for VS, use your compiler here + $ cmake ../src -G "Visual Studio 15 Win64" + $ cmake --build . --config Release + $ ./bin/greet + HelloA 0.2 Release + HelloB Release! + Greetings Release! + +But as explained above, the purpose of the lockfile is to capture the dependencies and use them later. +Let's pass the lockfile as an argument to guarantee the usage of the locked ``pkga/0.1@user/testing`` dependency: + +.. code-block:: bash + + $ conan install .. --lockfile=../locks/pkgb_deps.lock + > ... pkga/0.1@user/testing from local cache - Cache + $ cmake ../src -G "Visual Studio 15 Win64" + $ cmake --build . --config Release + $ ./bin/greet + HelloA 0.1 Release + HelloB Release! + Greetings Release! + +That's it. We managed to depend on ``pkga/0.1@user/testing`` instead of the ``pkga/0.2@user/testing`` although the later +satisfies the version range and is available in the cache. Using the same dependency was possible because we used the information stored in the lockfile. + + +Immutability +------------ + +A core concept of lockfiles is their immutability and the integrity of its data: + +.. important:: + + The information stored in a lockfile cannot be changed. Any attempt to modify locked data will result in + an error. + +For example, if now we try to do a :command:`conan install` that also builds ``pkga`` from source: + +.. code-block:: bash + + $ conan install .. --lockfile=../locks/pkgb_deps.lock --build=pkga + ERROR: Cannot build 'pkga/0.1@user/testing' because it is already locked in the input lockfile + +It is an error, because the ``pkga/0.1@user/testing`` dependency was fully locked. When the lockfile was created, the +``pkga/0.1@user/testing`` was found, including a binary, and that information was stored. Everytime this lockfile is +used, it assumes this package and binary exist and it will try to get them, but it will never allow to re-build, because +that can violate the integrity of the lockfile. For example, if we were using ``package_revision_mode``, a new binary +of ``pkga`` would produce new package-ids of all its consumers, that will not match the package-ids stored in the lockfile. + +It is possible though to control what is being locked with the ``--build`` argument provided to the :command:`conan lock create` +command. + +The same principle applies if we try to create a package for ``pkgb` and it tries to alter the user and channel ``user/testing`` +that were provided at the time of the :command:`conan lock create` command used above. + +.. code-block:: bash + + $ cd .. + $ conan create . user/stable --lockfile=locks/pkgb_deps.locked + ERROR: Attempt to modify locked pkgb/0.1@user/testing to pkgb/0.1@user/stable + +Again, it is important to keep the integrity. Package recipes can have conditional or parameterized dependencies, based on +user and channel for example. If we try to create the ``pkgb`` package with different user and channel, it could result in +a different dependency graph, totally incompatible with the one captured in the lockfile. If ``pkgb/0.1@user/testing`` was stored in +the lockfile, any command using this lockfile must respect and keep it without changes. + +.. note:: + + A package in a lockfile is fully locked if it contains a ``prev`` (package revision) field defined. + Fully locked packages cannot be built from sources. Partially locked packages do not contain a ``prev`` + defined. They lock the reference and the package-id, and they can be built from sources. + + +Reproducibility +--------------- + +That doesn't mean that a lockfile cannot evolve at all. Using the :command:`--lockfile` argument, we are able to create +``pkgb/0.1@user/testing`` guaranteeing it is being created depending on ``pkga/0.1@user/testing``. Additionally, if we use the +:command:`--lockfile-out` argument, we can obtain an updated version of the lockfile: + +.. code-block:: bash + + $ conan create . user/testing --lockfile=locks/pkgb_deps.lock --lockfile-out=locks/pkgb.lock + + +And if we inspect the new *locks/pkgb.lock* file: + +.. code-block:: text + + { + ... + "0": { + "ref": "pkgb/0.1@user/testing", + "options": "shared=False", + "package_id": "2418b211603ca0a3858d9dd1fc1108d54a4cab99", + "prev": "0", + "modified": true, + "requires": ["1"], + "context": "host" + } + ... + } + +It can be appreciated in *locks/pkgb.lock* that now ``pkgb/0.1@user/testing`` is fully locked, as a package (not a local *conanfile.py*), +and contains a ``package_id``. So if we try to use this new file for creating the package again, it will error, +as a package that is fully locked cannot be rebuilt: + + +.. code-block:: bash + + $ conan create . user/testing --lockfile=locks/pkgb.lock + ERROR: Attempt to modify locked pkgb/0.1@user/testing to pkgb/0.1@user/testing + + +But we can reproduce the same set of dependencies and the creation of ``pkgb``, using the *pkgb_deps.lock* lockfile: + +.. code-block:: bash + + $ conan create . user/testing --lockfile=locks/pkgb_deps.lock # OK + + +The *pkgb.lock* can be used later in time to install the ``pkgb`` application (the ``pkgb`` *conanfile.py* contains a ``deploy()`` +method for convenience for this example), and get the same package and dependencies: + +.. code-block:: bash + + $ cd .. + $ mkdir consume + $ cd consume + $ conan install pkgb/0.1@user/testing --lockfile=../pkgb/locks/pkgb.lock + $ ./bin/greet + HelloA 0.1 Release + HelloB Release! + Greetings Release! + +As long as we have the *pkgb.lock* lockfile, we will be able to robustly reproduce this install, even if the packages were +uploaded to a server, if there are new versions that satisfy the version ranges, etc. + + +.. important:: + + All the examples and documentation of this section is done with version ranges and revisions disabled. + Lockfiles also work and can lock both recipe and package revisions, with the same behavior as + version-ranges. All is necessary is to enable revisions. The only current limitation is that the local + cache cannot store more than one revision at a time, but that is a limitation of the cache and unrelated + to lockfiles.