-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: add a GitHub Action for generating new clients using the hermetic build scripts #10488
Changes from 1 commit
8cf7df0
74a3873
94119ae
7b19853
b8787c5
1bfd884
d13364e
affb741
72d54dc
6483d7c
309e4e0
e62f1e6
6a5172d
ca356c1
db13151
1a32e29
9f8e6f1
46620a3
124b4fd
c0a1b54
c7429c0
142fc18
aef76f2
eeabd19
5ef7f96
918875a
f71f2b4
78fda36
330f888
b964c4a
910e026
fedd456
d4a4909
d623ed5
26e4e32
df167fb
94fa787
4417f44
1eb08df
f10fbd6
8f489ab
da92cf1
81d9274
9afaf9f
03908a7
28da276
ce4125a
66872b3
12f7f95
9675bab
c4f426d
6e326f5
b544a21
b0bfbe6
b92c8d7
d118665
76fdeb5
42165b5
8b49863
19b5fac
fe525b5
94fea6f
0ff665c
4b9f9f2
cac37a7
45d929c
b4ecb26
0389090
3c977e8
c7e9404
fa469d0
d1eafd0
c3afa11
aae9d07
d3b2e57
62b2c7f
4c8bcfb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
name: Generate new GAPIC client library (Hermetic Build) | ||
on: | ||
workflow_dispatch: | ||
# some inputs are ommited due to limit of 10 input arguments | ||
inputs: | ||
api_shortname: | ||
required: true | ||
type: string | ||
description: "`api_shortname`: Name for the new directory name and (default) artifact name" | ||
name_pretty: | ||
required: true | ||
type: string | ||
description: "`name_pretty`: The human-friendly name that appears in README.md" | ||
proto_path: | ||
required: true | ||
type: string | ||
description: | | ||
`proto_path`: Path to proto file from the root of the googleapis repository to the | ||
directory that contains the proto files (without the version). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why not just use the versioned proto_path? I think it fits better since we can just add it as a GAPIC directly. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is to match the existing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was initially forcing the user to provide a versioned path until I saw the behavior of the old script There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we have to follow the same behavior, its a tool that is used only by our own team, if the new behavior means less code and no additional toil, then we should go for it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we have to follow the same behavior, its a tool that is used only by our own team, if the new behavior means less code and no additional toil, then we should go for it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed. As discussed, I modified the script to support versioned proto_paths only. |
||
For example, to generate the library for 'google/maps/routing/v2', | ||
then you specify this value as 'google/maps/routing' | ||
product_docs: | ||
required: true | ||
type: string | ||
description: "`product_docs`: Documentation URL that appears in README.md" | ||
rest_docs: | ||
required: false | ||
type: string | ||
description: | | ||
`rest_docs`: If it exists, link to the REST Documentation for a service | ||
rpc_docs: | ||
required: false | ||
type: string | ||
description: | | ||
`rpc_docs`: If it exists, link to the RPC Documentation for a service | ||
api_description: | ||
diegomarquezp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
required: true | ||
description: "`api_description`: Description that appears in README.md" | ||
transport: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't need to specify |
||
required: false | ||
type: choice | ||
default: grpc | ||
options: | ||
- grpc | ||
- http | ||
- both | ||
description: "`transport`: A label that appears in repo-metadata.json" | ||
destination_name: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't have a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
required: false | ||
type: string | ||
description: | | ||
`destination_name`: The directory name of the new library. By default it's | ||
java-<api_shortname> | ||
distribution_name: | ||
required: false | ||
type: string | ||
description: | | ||
`distribution_name`: Maven coordinates of the generated library. By default it's | ||
com.google.cloud:google-cloud-<api_shortname> | ||
|
||
jobs: | ||
generate: | ||
runs-on: ubuntu-22.04 | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: actions/setup-python@v4 | ||
with: | ||
python-version: '3.9' | ||
cache: 'pip' # caching pip dependencies | ||
- name: Install new-client.py dependencies | ||
run: pip install --require-hashes -r generation/new_client_hermetic_build/requirements.in | ||
- name: Add entry to generation_config.yaml | ||
diegomarquezp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
id: config_generation | ||
run: | | ||
set -x | ||
arguments="--api_shortname=\"${API_SHORTNAME}\" \ | ||
--proto-path=\"${PROTO_PATH}\" \ | ||
--name-pretty=\"${NAME_PRETTY}\" \ | ||
--product-docs=\"${PRODUCT_DOCS}\" \ | ||
--api-description=\"${API_DESCRIPTION}\"" | ||
|
||
# helper function that appends a python argument only if specified in the GH action inputs | ||
append_argument() { | ||
py_arg=$1 | ||
# env vars look exactly like new-client arguments but uppercase + underscores | ||
env_name=$(echo "${py_arg}" | sed 's/-/_/g' | sed -e 's/\([a-z]\)/\U\1/g') | ||
if [[ -n "${!env_name}" ]]; then | ||
# $(echo) is redundant but it works around a syntax highlighting problem in vim | ||
arguments=$(echo "${arguments} --${py_arg}=\"${!env_name}\"") | ||
fi | ||
} | ||
|
||
declare -a optional_args=('transport' 'destination-name' 'distribution-name' 'group-id' 'rest-docs' 'rpc-docs') | ||
|
||
for python_argument in "${optional_args[@]}"; do | ||
append_argument "${python_argument}" | ||
done | ||
echo "::set-output name=new_library_args::${arguments}" | ||
echo "${arguments} --googleapis-gen-url=\"${GOOGLEAPIS_GEN_URL}\"" \ | ||
| xargs python generation/new_client_hermetic_build/new-client.py generate | ||
env: | ||
GOOGLEAPIS_GEN_URL: https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_TOKEN }}@github.com/googleapis/googleapis-gen.git | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
NAME_PRETTY: ${{ github.event.inputs.name_pretty }} | ||
PROTO_PATH: ${{ github.event.inputs.proto_path }} | ||
PRODUCT_DOCS: ${{ github.event.inputs.product_docs }} | ||
REST_DOCS: ${{ github.event.inputs.rest_docs }} | ||
RPC_DOCS: ${{ github.event.inputs.rpc_docs }} | ||
API_DESCRIPTION: ${{ github.event.inputs.api_description }} | ||
TRANSPORT: ${{ github.event.inputs.transport }} | ||
DESTINATION_NAME: ${{ github.event.inputs.destination_name }} | ||
DISTRIBUTION_NAME: ${{ github.event.inputs.distribution_name }} | ||
- name: setup docker environment | ||
shell: bash | ||
run: | | ||
# we create a volume pointing to `pwd` (google-cloud-java) that will | ||
# be referenced by the container and its children | ||
if [[ $(docker volume inspect repo-google-cloud-java) != '[]' ]]; then | ||
docker volume rm repo-google-cloud-java | ||
fi | ||
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind" | ||
- name: generate from configuration | ||
shell: bash | ||
run: | | ||
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java" | ||
docker run --rm \ | ||
${repo_volumes} \ | ||
-v /tmp:/tmp \ | ||
-v /var/run/docker.sock:/var/run/docker.sock \ | ||
-e "RUNNING_IN_DOCKER=true" \ | ||
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \ | ||
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \ | ||
python /src/generate_repo.py generate \ | ||
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \ | ||
--repository-path=/workspace/google-cloud-java \ | ||
--target-library-api-shortname=${API_SHORTNAME} | ||
env: | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
- name: Push to branch and create PR | ||
run: | | ||
set -x | ||
[ -z "`git config user.email`" ] && git config --global user.email "cloud-java-bot@google.com" | ||
[ -z "`git config user.name`" ] && git config --global user.name "cloud-java-bot" | ||
|
||
# create and push to branch in origin | ||
# random_id allows multiple runs of this workflow | ||
random_id=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 5; echo) | ||
branch_name="new-library/${{ github.event.inputs.api_shortname }}-${random_id}" | ||
git checkout -b "${branch_name}" | ||
git add --all | ||
commit_message="feat: [${API_SHORTNAME}] new module for ${API_SHORTNAME}" | ||
git commit -m "${commit_message}" | ||
git remote add monorepo https://cloud-java-bot:${GH_TOKEN}@github.com/${{ github.repository }}.git | ||
git fetch -q --unshallow monorepo | ||
git push -f monorepo "${branch_name}" | ||
|
||
# create PR | ||
pr_body="Generated by @${USERNAME} via [generate_new_client_hermetic_build.yaml](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml) | ||
|
||
Command used: | ||
|
||
\`\`\` | ||
python generation/new_client_hermetic_build/new-client.py generate ${GENERATION_ARGUMENTS} | ||
\`\`\`" | ||
gh pr create --title "${commit_message}" --label "owlbot:run" --head "${branch_name}" --body "${pr_body}" | ||
env: | ||
USERNAME: ${{ github.actor }} | ||
API_SHORTNAME: ${{ github.event.inputs.api_shortname }} | ||
GENERATION_ARGUMENTS: ${{ steps.config_generation.outputs.new_library_args }} | ||
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is an existing limitation of Github actions, but IMO this is a indication that Github action may not be a good fit for new client library generation. Because we are clearly omitting some parameters here, they may not be "required" to generate a new library, but
repo_metadata.json
would be missing some info without those parameters.I think we should find a different way sooner than later, but it would be out of scope for this PR. cc: @suztomo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are missing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, these four in repo_metadata.json, we might be able to get them heuristically, but if we want to override them, there is no way other than manually adding them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we discussed today, the long term solution is to
CC: @JoeWang1127
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the example.
The best way is to automatically get the data from the service config yaml.