-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: add a GitHub Action for generating new clients using the hermetic build scripts #10488
Merged
diegomarquezp
merged 77 commits into
googleapis:main
from
diegomarquezp:new-library-hermetic-build
Mar 19, 2024
Merged
Changes from 74 commits
Commits
Show all changes
77 commits
Select commit
Hold shift + click to select a range
8cf7df0
chore: add new-client for hermetic build
diegomarquezp 74a3873
enable yaml insertion
diegomarquezp 94119ae
normalize yaml format
diegomarquezp 7b19853
add requirements file
diegomarquezp b8787c5
add gh action
diegomarquezp 1bfd884
add pull_request event for testing
diegomarquezp d13364e
add requirements.txt
diegomarquezp affb741
use requirements with hashes
diegomarquezp 72d54dc
add requirement hashes
diegomarquezp 6483d7c
fix requirements.in
diegomarquezp 309e4e0
remove testing event
diegomarquezp e62f1e6
fix requirement hashes
diegomarquezp 6a5172d
fix generation script call
diegomarquezp ca356c1
remove usage of googleapis-gen-url
diegomarquezp db13151
fix gapic entry generation
diegomarquezp 1a32e29
add debug print statement
diegomarquezp 9f8e6f1
remove mistakenly added chat library
diegomarquezp 46620a3
suppress debug output
diegomarquezp 124b4fd
fix pr message, fix pr label
diegomarquezp c0a1b54
fix pr description message
diegomarquezp c7429c0
improve pr descriptoin
diegomarquezp 142fc18
fix comment
diegomarquezp aef76f2
add readme
diegomarquezp eeabd19
add advanced options
diegomarquezp 5ef7f96
fix syntax
diegomarquezp 918875a
clarify type of workflow
diegomarquezp f71f2b4
fix advanced options
diegomarquezp 78fda36
checkout latest fix in main branch
diegomarquezp 330f888
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp b964c4a
remove transport form workflow options
diegomarquezp 910e026
change destination_name to library_name, remove cloud_api
diegomarquezp fedd456
remove library existence check
diegomarquezp d4a4909
change product_docs to product_documentation
diegomarquezp d623ed5
Revert "change product_docs to product_documentation"
diegomarquezp 26e4e32
add api_reference
diegomarquezp df167fb
add codeowner_team
diegomarquezp 94fa787
add excluded_dependencies
diegomarquezp 4417f44
add excluded_poms
diegomarquezp 1eb08df
add googleapis_commitish
diegomarquezp f10fbd6
add mutually exclusive logic for group_id and distribution_name
diegomarquezp 8f489ab
add issue_tracker
diegomarquezp da92cf1
add extra_versioned_modules
diegomarquezp 81d9274
remove untracked file
diegomarquezp 9afaf9f
improve logic for inferring yaml variables
diegomarquezp 03908a7
remove debug file
diegomarquezp 28da276
compute all proto_path verions
diegomarquezp ce4125a
update googleapis_commitish for testing
diegomarquezp 66872b3
add cleanup of googleapis folder
diegomarquezp 12f7f95
add xtrace
diegomarquezp 9675bab
update dependencies
diegomarquezp c4f426d
update dependencies ii
diegomarquezp 6e326f5
remove owlbot label
diegomarquezp b544a21
add owlbot run label
diegomarquezp b0bfbe6
fix library_name
diegomarquezp b92c8d7
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp d118665
restore label
diegomarquezp 76fdeb5
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp 42165b5
reorganize required params
diegomarquezp 8b49863
check if library is releasable
diegomarquezp 19b5fac
sync generation_config.yaml
diegomarquezp fe525b5
temporarily use latest commitish
diegomarquezp 94fea6f
Revert "temporarily use latest commitish"
diegomarquezp 0ff665c
add python script to parse arguments
diegomarquezp 4b9f9f2
fix argument generation
diegomarquezp cac37a7
fix api-shortname parameter
diegomarquezp 45d929c
restart tests
diegomarquezp b4ecb26
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp 0389090
fix documentation and comments
diegomarquezp 3c977e8
add `library_name` explanation
diegomarquezp c7e9404
clarify api_shortname
diegomarquezp fa469d0
rename to add-new-client-config
diegomarquezp d1eafd0
remove redundant library_name assignment
diegomarquezp c3afa11
support for versioned proto_paths only
diegomarquezp aae9d07
instructions for multiple proto_paths
diegomarquezp d3b2e57
restore config yaml
diegomarquezp 62b2c7f
restore generation config
diegomarquezp 4c8bcfb
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
157 changes: 157 additions & 0 deletions
157
.github/workflows/generate_new_client_hermetic_build.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,157 @@ | ||
name: Generate new GAPIC client library (Hermetic Build) | ||
on: | ||
workflow_dispatch: | ||
# some inputs are ommited due to limit of 10 input arguments | ||
inputs: | ||
api_shortname: | ||
required: true | ||
type: string | ||
description: "`api_shortname`: Name for the new directory name and (default) artifact name" | ||
name_pretty: | ||
required: true | ||
type: string | ||
description: "`name_pretty`: The human-friendly name that appears in README.md" | ||
api_description: | ||
required: true | ||
description: "`api_description`: Description that appears in README.md" | ||
proto_path: | ||
required: true | ||
type: string | ||
description: | | ||
`proto_path`: Path to proto file from the root of the googleapis repository to the | ||
directory that contains the proto files (with the version). | ||
For example, to generate `v2` of google/cloud/library, | ||
you must pass google/cloud/library/v2 | ||
product_docs: | ||
required: true | ||
type: string | ||
description: "`product_docs`: Documentation URL that appears in README.md" | ||
rest_docs: | ||
required: false | ||
type: string | ||
description: | | ||
`rest_docs`: If it exists, link to the REST Documentation for a service | ||
rpc_docs: | ||
required: false | ||
type: string | ||
description: | | ||
`rpc_docs`: If it exists, link to the RPC Documentation for a service | ||
library_name: | ||
required: false | ||
type: string | ||
description: | | ||
`library_name`: The directory name of the new library. By default it's | ||
java-<api_shortname> | ||
distribution_name: | ||
required: false | ||
type: string | ||
description: | | ||
`distribution_name`: Maven coordinates of the generated library. By default it's | ||
com.google.cloud:google-cloud-<api_shortname> | ||
|
||
jobs: | ||
generate: | ||
runs-on: ubuntu-22.04 | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: actions/setup-python@v4 | ||
with: | ||
python-version: '3.9' | ||
cache: 'pip' # caching pip dependencies | ||
- name: Install add-new-client-config.py dependencies | ||
run: pip install --require-hashes -r generation/new_client_hermetic_build/requirements.txt | ||
- name: Add entry to generation_config.yaml | ||
diegomarquezp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
id: config_generation | ||
run: | | ||
set -x | ||
arguments=$(python generation/new_client_hermetic_build/generate-arguments.py) | ||
echo "::set-output name=new_library_args::${arguments}" | ||
echo "${arguments}" \ | ||
| xargs python generation/new_client_hermetic_build/add-new-client-config.py add-new-library | ||
env: | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
NAME_PRETTY: ${{ github.event.inputs.name_pretty }} | ||
PROTO_PATH: ${{ github.event.inputs.proto_path }} | ||
PRODUCT_DOCS: ${{ github.event.inputs.product_docs }} | ||
REST_DOCS: ${{ github.event.inputs.rest_docs }} | ||
RPC_DOCS: ${{ github.event.inputs.rpc_docs }} | ||
API_DESCRIPTION: ${{ github.event.inputs.api_description }} | ||
LIBRARY_NAME: ${{ github.event.inputs.library_name }} | ||
DISTRIBUTION_NAME: ${{ github.event.inputs.distribution_name }} | ||
- name: setup docker environment | ||
shell: bash | ||
run: | | ||
set -x | ||
# we create a volume pointing to `pwd` (google-cloud-java) that will | ||
# be referenced by the container and its children | ||
if [[ $(docker volume inspect repo-google-cloud-java) != '[]' ]]; then | ||
docker volume rm repo-google-cloud-java | ||
fi | ||
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind" | ||
- name: generate from configuration | ||
id: generation | ||
shell: bash | ||
run: | | ||
set -x | ||
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java" | ||
echo "::set-output name=repo_volumes::${repo_volumes}" | ||
docker run --rm \ | ||
${repo_volumes} \ | ||
-v /tmp:/tmp \ | ||
-v /var/run/docker.sock:/var/run/docker.sock \ | ||
-e "RUNNING_IN_DOCKER=true" \ | ||
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \ | ||
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \ | ||
python /src/generate_repo.py generate \ | ||
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \ | ||
--repository-path=/workspace/google-cloud-java \ | ||
--target-library-api-shortname=${API_SHORTNAME} | ||
env: | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
- name: Push to branch and create PR | ||
run: | | ||
set -x | ||
[ -z "`git config user.email`" ] && git config --global user.email "cloud-java-bot@google.com" | ||
[ -z "`git config user.name`" ] && git config --global user.name "cloud-java-bot" | ||
|
||
# create and push to branch in origin | ||
# random_id allows multiple runs of this workflow | ||
random_id=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 5; echo) | ||
branch_name="new-library/${{ github.event.inputs.api_shortname }}-${random_id}" | ||
git checkout -b "${branch_name}" | ||
git add --all | ||
commit_message="feat: [${API_SHORTNAME}] new module for ${API_SHORTNAME}" | ||
git commit -m "${commit_message}" | ||
git remote add monorepo https://cloud-java-bot:${GH_TOKEN}@github.com/${{ github.repository }}.git | ||
git fetch -q --unshallow monorepo | ||
git push -f monorepo "${branch_name}" | ||
|
||
# create PR | ||
pr_body="Generated by @${USERNAME} via [generate_new_client_hermetic_build.yaml](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml) | ||
|
||
Command used: | ||
|
||
\`\`\` | ||
python generation/new_client_hermetic_build/add-new-client-config.py add-new-client ${GENERATION_ARGUMENTS} | ||
|
||
docker run --rm \\ | ||
${DOCKER_VOLUMES} \\ | ||
-v /tmp:/tmp \\ | ||
-v /var/run/docker.sock:/var/run/docker.sock \\ | ||
-e \"RUNNING_IN_DOCKER=true\" \\ | ||
-e \"REPO_BINDING_VOLUMES=${DOCKER_VOLUMES}\" \\ | ||
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \\ | ||
python /src/generate_repo.py generate \\ | ||
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \\ | ||
--repository-path=/workspace/google-cloud-java \\ | ||
--target-library-api-shortname=${API_SHORTNAME} | ||
|
||
\`\`\`" | ||
gh pr create --title "${commit_message}" --label "owlbot:run" --head "${branch_name}" --body "${pr_body}" | ||
env: | ||
USERNAME: ${{ github.actor }} | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
GENERATION_ARGUMENTS: ${{ steps.config_generation.outputs.new_library_args }} | ||
DOCKER_VOLUMES: ${{ steps.generation.outputs.repo_volumes }} | ||
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
# New client generation (GitHub Action) | ||
This new generation workflow enables generation of new libraries by | ||
1. appending a new library to our [generation_config.yaml](https://github.com/googleapis/google-cloud-java/blob/c7429c0eec419c01d4e2fe14d063b9335efb810b/generation_config.yaml). | ||
2. running the hermetic build scripts docker image and | ||
generate the newly added library. | ||
3. create a PR with the changes. | ||
|
||
|
||
## Components | ||
### `new_library_hermetic_build/add-new-client-config.py` | ||
This script takes 10 arguments that map to items in the newly added library that | ||
goes in `generation_config.yaml`. A new entry will be added to `libraries` with | ||
the necessary generation configuration. | ||
|
||
### `.github/workflows/generate_new_client_hermetic_build.yaml` | ||
This workflow orchestrates the `add-new-client-config.py` script and also uses our docker | ||
image | ||
([gcr.io/cloud-devrel-public-resources/java-library-generation](https://pantheon.corp.google.com/gcr/images/cloud-devrel-public-resources/global/java-library-generation)) | ||
to generate a new library. It also creates the pull request. | ||
|
||
|
||
## Execute the Github Action | ||
|
||
In order to run the | ||
[Github Action](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml) | ||
, you need to specify a few parameters. | ||
These parameters will be available in the Cloud Drop link (a YAML file) included in the buganizer request. | ||
The example in this README uses AlloyDB's [Cloud Drop](https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1/alloydb_v1.yaml) file as an example. | ||
|
||
### API short name (`api_shortname`) | ||
|
||
As a convenience for the subsequent commands, we need an identifier for the | ||
library, called `api_shortname`. | ||
This identifier will be used by default to generate the following: | ||
* `--distribution-name` | ||
* `--library_name` | ||
|
||
The corresponding value in the Cloud Drop page is `api_short_name`. | ||
|
||
Example: `alloydb` | ||
|
||
> [!IMPORTANT] | ||
> `api_short_name` is not always unique across client libraries. | ||
> In the instance that the `api_short_name` is already in use by an existing | ||
> client library, you will need to determine a unique name OR to pass a unique | ||
> `library_name`. | ||
> See [Advanced Options](#advanced-options). | ||
|
||
### Proto path (`proto_path`) | ||
|
||
This is the path from the internal `google3/third_party/googleapis/stable` root to the | ||
directory that contains the proto definitions for a specific version. | ||
For example: `google/datastore/v2`. Root-level proto paths like | ||
`google/datastore` are not supported. | ||
Note that the internal `google3/third_party/googleapis/stable` directory is mirrored externally in https://github.com/googleapis/googleapis/blob/master/. | ||
|
||
For example, if the buganizer ticket includes: | ||
|
||
> Link to protos: `http://...(omit).../google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. | ||
|
||
then the corresponding external mirrored proto is here: `https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml`. | ||
|
||
Therefore, the "proto path" value we supply to the command is | ||
`google/cloud/alloydb/v1alpha`. | ||
|
||
We will publish a single module for a service that includes the specified version | ||
(in the example, `v1alpha`). Any future version must be manually added to | ||
the configuration yaml (`google-cloud-java/generation_config.yaml`) | ||
|
||
#### More than one `proto_path` | ||
|
||
If you need another `proto_path` in the library, you must manually add it | ||
to `google-cloud-java/generation_config.yaml` after generating the new client. | ||
|
||
### Name pretty (`name_pretty`) | ||
|
||
The corresponding value in the Cloud Drop page is `title`. | ||
|
||
Example: `AlloyDB API` | ||
|
||
### Product Docs (`product_docs`) | ||
|
||
The corresponding value in the Cloud Drop page is `documentation_uri`. | ||
The value must starts with "https://". | ||
|
||
Example: `https://cloud.google.com/alloydb/docs` | ||
|
||
### REST Docs (`rest_docs`) | ||
|
||
The corresponding value in the Cloud Drop page is `rest_reference_documentation_uri`. | ||
The value must starts with "https://". | ||
|
||
Example: `https://cloud.google.com/alloydb/docs/reference/rest` | ||
|
||
If the value exists, add it as a flag to the python command below (see [Advanced | ||
Options](#advanced-options]): | ||
`--rest-docs="https://cloud.google.com/alloydb/docs/reference/rest" \` | ||
|
||
### RPC Docs (`rpc_docs`) | ||
|
||
The corresponding value in the Cloud Drop page is `proto_reference_documentation_uri`. | ||
The value must starts with "https://". | ||
|
||
Example: `https://cloud.google.com/speech-to-text/docs/reference/rpc` | ||
|
||
If the value exists, add it as a flag to the python command below (see [Advanced | ||
Options](#advanced-options]): | ||
`--rpc-docs="https://cloud.google.com/speech-to-text/docs/reference/rpc" \` | ||
|
||
### API description (`api_description`) | ||
|
||
The corresponding value in the Cloud Drop page is `documentation.summary` or `documentation.overview`. | ||
If both of those fields are missing, take the description from the product page above. Use the first sentence to keep it concise. | ||
|
||
Example: | ||
``` | ||
AlloyDB for PostgreSQL is an open source-compatible database service that | ||
provides a powerful option for migrating, modernizing, or building | ||
commercial-grade applications. | ||
``` | ||
|
||
### Distribution Name (`distribution_name`) | ||
|
||
This variable determines the Maven coordinates of the generated library. It | ||
defaults to `com.google.cloud:google-cloud-{api_shortname}`. This mainly affect | ||
the values in the generated `pom.xml` files. | ||
|
||
### Library Name (`library_name`) | ||
|
||
This variable indicates the output folder of the library. For example you can | ||
have two libraries with `alloydb` (AlloyDB and AlloyDB Connectors) | ||
as `api_shortname`. In order to avoid both | ||
libraries going to the default `java-alloydb` folder, we can override this | ||
behavior by specifying a value like `alloydb-connectors` so the AlloyDB | ||
Connectors goes to `java-alloydb-connectors`. | ||
|
||
## Advanced Options | ||
|
||
In case the steps above don't show you how to specify the desired options, you can | ||
run the `add-new-client-config.py` script in your local evironment. The advanced options | ||
not shown in the section above **cannot be specified in the Github Action**, | ||
JoeWang1127 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
hence the need for a local run (refer to the "Prerequisites | ||
(for local environment)" section). | ||
For the explanation of the available parameters, run: | ||
`python3.9 generation/new_client_hermetic_build/add-new-client-config.py generate --help`. | ||
|
||
After you run the script, you will see that the `generation_config.yaml` file | ||
was modified (or the script exited because the library already existed) | ||
|
||
The last step you need is to `cd` into the root of `google-cloud-java` and run | ||
``` | ||
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind" | ||
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java" | ||
docker run --rm \ | ||
${repo_volumes} \ | ||
-v /tmp:/tmp \ | ||
-v /var/run/docker.sock:/var/run/docker.sock \ | ||
-e "RUNNING_IN_DOCKER=true" \ | ||
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \ | ||
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \ | ||
python /src/generate_repo.py generate \ | ||
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \ | ||
--repository-path=/workspace/google-cloud-java \ | ||
--target-library-api-shortname=<your api_shortname> | ||
|
||
``` | ||
|
||
This docker container will run the generation scripts and generate the library | ||
in your repo. You can create a PR explaining what commands you used (the docker | ||
command is not as informative as the `add-new-client-config.py` call, so make sure at least | ||
the add-new-client-config.py arguments were listed). |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is an existing limitation of Github actions, but IMO this is a indication that Github action may not be a good fit for new client library generation. Because we are clearly omitting some parameters here, they may not be "required" to generate a new library, but
repo_metadata.json
would be missing some info without those parameters.I think we should find a different way sooner than later, but it would be out of scope for this PR. cc: @suztomo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are missing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, these four in repo_metadata.json, we might be able to get them heuristically, but if we want to override them, there is no way other than manually adding them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we discussed today, the long term solution is to
CC: @JoeWang1127
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the example.
The best way is to automatically get the data from the service config yaml.