Move "prepare-provider-documentation" to Breeze #35586
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
kaxil,
pierrejeambrun,
ashb and
jedcunningham
as code owners
|
I know there is a lot of code moved here, but I think the easiest way to review (except the regular - code quality) is to see result of this one run. Anyone should be able review the output of CI command in the "Prepare providers" job. The way how it is run in CI is that it "pretends" you are a relese manager and perform release documentation preparattion, randomly making decisions on the type of change (Breaking/Feature/Bugfix/Doc-only) and in the output you will see the generated documentation and the output of such process run. Also you can do this manually to check it:
@eladkal -> You might appreciate that I fixed the few annoying things in this PR:
@ashb -> I am finally cleaning it up, this is somethging you - rightfully - complained about in the past that we had this historically weird After this (and few follow-up PR for package preparation - much smaller than this one) , all the logic for provider release will be fiinally in single place (breeze) and it will be easier to debug and maintain. |
potiuk
force-pushed
the
move-prepare-providers-documentation-to-breeze
branch
5 times, most recently
from
01c0e1b
to
d9d2dda
Compare
|
All right . I think that one is already going to be green... One example for easier review (but there are more in the CI job output): https://github.com/apache/airflow/actions/runs/6842478060/job/18604229396?pr=35586 AirbyteBumping patchlevel: 
Commits: 
Changelog: 
|
potiuk
force-pushed
the
move-prepare-providers-documentation-to-breeze
branch
from
d9d2dda
to
547cd12
Compare
|
Looks green now :). |
potiuk
changed the title
potiuk
force-pushed
the
move-prepare-providers-documentation-to-breeze
branch
from
547cd12
to
fddf030
Compare
potiuk
changed the title
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 13, 2023
This is a follow-up after apache#35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
|
Side effect-> preparing provider documentation goes down from 14 minutes to 3 |
hussein-awala
approved these changes
Nov 14, 2023
potiuk
force-pushed
the
move-prepare-providers-documentation-to-breeze
branch
from
844b297
to
c261c23
Compare
This PR moves the functionality of preparing provider documentation from a python script inside Breeze CI image to breeze Python package. This is the first of the series of moves that will simplify the way how provider packages are being built and prepared with the aim of improving security of the supply chain and make it easier to debug and modify the release process. Historically, release process has been run inside of Breeze for several reasons: isolation of running package preparation from Host environment, the need to keep separate virtualenv and because we run verification of provider packages during release process - which requires the CI environment with all its dependencies. So far the process looked like this: * bash breeze parsed the arguments * bash breeze started the docker bash script with packages as parameters * the bash script in CI image looped over the packages and run python prepare_provider_packages.py (twice) to generate docs and update changelog (this is interactive process where release manager makes decision on bumping versions). Those python script performed verification on provider.yaml files * the bash script summarized the packages and displayed status of preparation However after moving to Python based breeze, we can simplify it all and run all those steps in Python breeze internal code - no need to go to docker and use bash scripts. We also do not have to do verification of provider.yaml files is done already extensively in pre-commit. This PR moves all this logic to inside Breeze. There is stil remainig duplicated code in the original in-container `prepare_provider_packages.py` remaining, this duplication will be removed by subsequent PRs where other release management commands for provider packages will also be moved to Breeze as follow-up of this PR. This PR has the following changes: * move the provider decumentation code from `dev/provider_packages` to `dev/breeze/` (and from in-container to in-breeze-venv execution) * completely removed the intermediate bash script and calling Python scripts from it - moving the logic to Breeze entirely * added better diagnostics of what happens when packages are classified with particular types of changes (added `special` style to show it) * cleaned and clarified `prepare-provider-documentation` commmand line flags * introduce explicit "non-interactive" mode that is used to run and test the command in CI and to test it locally * replace str with Path were files were used in the moved code * add unit tests covering unit-testable parts of the moved code * refactore the moved code to use utils available in Breeze * split the code into packages and versions (reusable utils) and specific code for preparing package documentation * cached provider.yaml information retrieved from providers * move provider documentation templates to Breeze * better error handling - they are now regular exceptions in Python process rather than interpreting the exit_codes passed from python sub-scripts returnig the codes to Bash * when release manager classifies package, only relevant sections are generated (Features/Breaking changes) based on decision and changes are automatically "guessed" only if release manager chose the section where they would fall in
potiuk
force-pushed
the
move-prepare-providers-documentation-to-breeze
branch
from
c261c23
to
9416085
Compare
Used the opportunity of jsonschema fix rebase to apply those 👍 |
potiuk
added a commit
that referenced
this pull request
Nov 15, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
that referenced
this pull request
Nov 15, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
that referenced
this pull request
Nov 15, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Nov 16, 2023
This PR fixes some of the new templates added in apache#35586 and reapplies them accross the board for all providers. There were a few small issues with those templates: * wrong location of the original template in the comment * extra comment and lines in the INDEX template * changes in __init__.py with shorter warning and avoiding "too long line" have been reapplied to all providers
potiuk
added a commit
that referenced
this pull request
Nov 16, 2023
This PR fixes some of the new templates added in #35586 and reapplies them accross the board for all providers. There were a few small issues with those templates: * wrong location of the original template in the comment * extra comment and lines in the INDEX template * changes in __init__.py with shorter warning and avoiding "too long line" have been reapplied to all providers
potiuk
added a commit
that referenced
this pull request
Nov 16, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
that referenced
this pull request
Nov 16, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
that referenced
this pull request
Nov 18, 2023
This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
potiuk
added a commit
that referenced
this pull request
Nov 18, 2023
…35617) This is a follow-up after #35586 and it depends on this one. It moves the whole functionality of preparing provider packages to breeze, removing the need of doing it in the Breeze CI image. Since we have Python breeze with its own environment managed via `pipx` we can now make sure that all the necessary packages are installed in this environment and run package building in the same environment Breeze uses. Previously we have been running all the package building inside the CI image for two reasons: * we could rely on the same version of build tools (wheel/setuptools) being installed in the CI image * security of the provider package preparation that used setuptools pre PEP-517 way of building packages that executed setup.py code In order to isolate execution of potentially arbitrary code in setup.py from the HOST environment in CI - where the host environment might have access to secrets and tokens that would allow it to break out of the sandbox for PRs coming from forks. The setup.py file has been prepared by breeze using JINJA templates but it was potentially possible to manipulate provider package directory structure and get "Python" injection into generated setup.py, so it was safer to run it in the isolated Breeze CI environment. This PR makes it secure to run it in the Host environment, because instead of generating setup.cfg and setup.py we generate pyproject.toml with all the necessary information and we are using PEP-517 compliant way of building provider packages - no arbitrary code executed via setup.py is possible this way on the host, so we can safely build provider packages in the host. We are generating declarative pyproject.toml for that rather than imperative setup.py, so we are safe to run the build process in the host without being afraid of executing arbitrary code. We are using flit as build tool - this is one of the popular build tools - created by Python Packaging team. It is simple and not too opinionated, it supports PEP-517 as well as PEP-621, so most of the project mnetadata in pyproject toml can be added to PEP-621 compliant "project" section of pyproject.toml. Together with the change we improves the process of generation of the extracted sources for the providers. Originally we copied the whole sources of Airflow to a single directory (provider_packages) and run sequentially provider packages building from that single directory, however it made it impossible to parallelise such builds - all providers had to be built sequentially. We change the approach now - instead of copying all airflow sources once to the single directory, we build providers in separate subdirectories of files/provider_packages/PROVIDER_ID and we only copy there relevant sources (i.e. only provider's subfolder from the "airflow/providers". This is quite a bit faster (each provider only gets built using only its own sources so just scanning the directory is faster) but it also allows to run package preparation in parallel because each provider is fully isolated from others. This PR also excludes not-needed `prepare_providers_package.py` and unneded `provider_packages` folder used to prepare providers before as well as bash script to build the providers and some unused bash functions.
ephraimbuddy
added
the
changelog:skip
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR moves the functionality of preparing provider documentation from a python script inside Breeze CI image to breeze Python package.
This is the first of the series of moves that will simplify the way how provider packages are being built and prepared with the aim of improving security of the supply chain and make it easier to debug and modify the release process.
Historically, release process has been run inside of Breeze for several reasons: isolation of running package preparation from Host environment, the need to keep separate virtualenv and because we run verification of provider packages during release process - which requires the CI environment with all its dependencies.
So far the process looked like this:
However after moving to Python based breeze, we can simplify it all and run all those steps in Python breeze internal code - no need to go to docker and use bash scripts. We also do not have to do verification of provider.yaml files is done already extensively in pre-commit.
This PR moves all this logic to inside Breeze.
There is stil remainig duplicated code in the original in-container
prepare_provider_packages.pyremaining, this duplication will be removed by subsequent PRs where other release management commands for provider packages will also be moved to Breeze as follow-up of this PR.This PR has the following changes:
dev/provider_packagestodev/breeze/(and from in-container to in-breeze-venv execution)specialstyle to show it)prepare-provider-documentationcommmand line flags^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.