Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add a GitHub Action for generating new clients using the hermetic build scripts #10488

Merged
Merged
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
8cf7df0
chore: add new-client for hermetic build
diegomarquezp Mar 5, 2024
74a3873
enable yaml insertion
diegomarquezp Mar 6, 2024
94119ae
normalize yaml format
diegomarquezp Mar 6, 2024
7b19853
add requirements file
diegomarquezp Mar 6, 2024
b8787c5
add gh action
diegomarquezp Mar 6, 2024
1bfd884
add pull_request event for testing
diegomarquezp Mar 6, 2024
d13364e
add requirements.txt
diegomarquezp Mar 6, 2024
affb741
use requirements with hashes
diegomarquezp Mar 6, 2024
72d54dc
add requirement hashes
diegomarquezp Mar 6, 2024
6483d7c
fix requirements.in
diegomarquezp Mar 6, 2024
309e4e0
remove testing event
diegomarquezp Mar 6, 2024
e62f1e6
fix requirement hashes
diegomarquezp Mar 6, 2024
6a5172d
fix generation script call
diegomarquezp Mar 6, 2024
ca356c1
remove usage of googleapis-gen-url
diegomarquezp Mar 6, 2024
db13151
fix gapic entry generation
diegomarquezp Mar 6, 2024
1a32e29
add debug print statement
diegomarquezp Mar 6, 2024
9f8e6f1
remove mistakenly added chat library
diegomarquezp Mar 6, 2024
46620a3
suppress debug output
diegomarquezp Mar 6, 2024
124b4fd
fix pr message, fix pr label
diegomarquezp Mar 7, 2024
c0a1b54
fix pr description message
diegomarquezp Mar 7, 2024
c7429c0
improve pr descriptoin
diegomarquezp Mar 7, 2024
142fc18
fix comment
diegomarquezp Mar 7, 2024
aef76f2
add readme
diegomarquezp Mar 7, 2024
eeabd19
add advanced options
diegomarquezp Mar 7, 2024
5ef7f96
fix syntax
diegomarquezp Mar 7, 2024
918875a
clarify type of workflow
diegomarquezp Mar 7, 2024
f71f2b4
fix advanced options
diegomarquezp Mar 7, 2024
78fda36
checkout latest fix in main branch
diegomarquezp Mar 7, 2024
330f888
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp Mar 7, 2024
b964c4a
remove transport form workflow options
diegomarquezp Mar 7, 2024
910e026
change destination_name to library_name, remove cloud_api
diegomarquezp Mar 7, 2024
fedd456
remove library existence check
diegomarquezp Mar 7, 2024
d4a4909
change product_docs to product_documentation
diegomarquezp Mar 7, 2024
d623ed5
Revert "change product_docs to product_documentation"
diegomarquezp Mar 7, 2024
26e4e32
add api_reference
diegomarquezp Mar 7, 2024
df167fb
add codeowner_team
diegomarquezp Mar 7, 2024
94fa787
add excluded_dependencies
diegomarquezp Mar 7, 2024
4417f44
add excluded_poms
diegomarquezp Mar 7, 2024
1eb08df
add googleapis_commitish
diegomarquezp Mar 7, 2024
f10fbd6
add mutually exclusive logic for group_id and distribution_name
diegomarquezp Mar 7, 2024
8f489ab
add issue_tracker
diegomarquezp Mar 7, 2024
da92cf1
add extra_versioned_modules
diegomarquezp Mar 7, 2024
81d9274
remove untracked file
diegomarquezp Mar 7, 2024
9afaf9f
improve logic for inferring yaml variables
diegomarquezp Mar 7, 2024
03908a7
remove debug file
diegomarquezp Mar 8, 2024
28da276
compute all proto_path verions
diegomarquezp Mar 8, 2024
ce4125a
update googleapis_commitish for testing
diegomarquezp Mar 8, 2024
66872b3
add cleanup of googleapis folder
diegomarquezp Mar 8, 2024
12f7f95
add xtrace
diegomarquezp Mar 8, 2024
9675bab
update dependencies
diegomarquezp Mar 8, 2024
c4f426d
update dependencies ii
diegomarquezp Mar 8, 2024
6e326f5
remove owlbot label
diegomarquezp Mar 11, 2024
b544a21
add owlbot run label
diegomarquezp Mar 11, 2024
b0bfbe6
fix library_name
diegomarquezp Mar 11, 2024
b92c8d7
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp Mar 11, 2024
d118665
restore label
diegomarquezp Mar 11, 2024
76fdeb5
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp Mar 11, 2024
42165b5
reorganize required params
diegomarquezp Mar 11, 2024
8b49863
check if library is releasable
diegomarquezp Mar 11, 2024
19b5fac
sync generation_config.yaml
diegomarquezp Mar 11, 2024
fe525b5
temporarily use latest commitish
diegomarquezp Mar 11, 2024
94fea6f
Revert "temporarily use latest commitish"
diegomarquezp Mar 11, 2024
0ff665c
add python script to parse arguments
diegomarquezp Mar 11, 2024
4b9f9f2
fix argument generation
diegomarquezp Mar 11, 2024
cac37a7
fix api-shortname parameter
diegomarquezp Mar 11, 2024
45d929c
restart tests
diegomarquezp Mar 11, 2024
b4ecb26
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp Mar 12, 2024
0389090
fix documentation and comments
diegomarquezp Mar 12, 2024
3c977e8
add `library_name` explanation
diegomarquezp Mar 12, 2024
c7e9404
clarify api_shortname
diegomarquezp Mar 12, 2024
fa469d0
rename to add-new-client-config
diegomarquezp Mar 13, 2024
d1eafd0
remove redundant library_name assignment
diegomarquezp Mar 13, 2024
c3afa11
support for versioned proto_paths only
diegomarquezp Mar 13, 2024
aae9d07
instructions for multiple proto_paths
diegomarquezp Mar 13, 2024
d3b2e57
restore config yaml
diegomarquezp Mar 13, 2024
62b2c7f
restore generation config
diegomarquezp Mar 18, 2024
4c8bcfb
Merge remote-tracking branch 'origin/main' into new-library-hermetic-…
diegomarquezp Mar 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add gh action
  • Loading branch information
diegomarquezp committed Mar 6, 2024
commit b8787c58d24ee89a08647068939727ce93b51f1b
171 changes: 171 additions & 0 deletions .github/workflows/generate_new_client_hermetic_build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
name: Generate new GAPIC client library (Hermetic Build)
on:
workflow_dispatch:
# some inputs are ommited due to limit of 10 input arguments
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is an existing limitation of Github actions, but IMO this is a indication that Github action may not be a good fit for new client library generation. Because we are clearly omitting some parameters here, they may not be "required" to generate a new library, but repo_metadata.json would be missing some info without those parameters.
I think we should find a different way sooner than later, but it would be out of scope for this PR. cc: @suztomo

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

repo_metadata.json would be missing some info without those parameters.

What are missing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, these four in repo_metadata.json, we might be able to get them heuristically, but if we want to override them, there is no way other than manually adding them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed today, the long term solution is to

  1. Make the PR generation logic more generic so that it can recognize any changes to the generation config
  2. Trigger the generation process on every generation config change.
  3. Create a PR(Manually or still use this Github action) that adds a new section for the new client, which should trigger the process above.
    CC: @JoeWang1127

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the example.

manually adding them.

The best way is to automatically get the data from the service config yaml.

inputs:
api_shortname:
required: true
type: string
description: "`api_shortname`: Name for the new directory name and (default) artifact name"
name_pretty:
required: true
type: string
description: "`name_pretty`: The human-friendly name that appears in README.md"
proto_path:
required: true
type: string
description: |
`proto_path`: Path to proto file from the root of the googleapis repository to the
directory that contains the proto files (without the version).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just use the versioned proto_path? I think it fits better since we can just add it as a GAPIC directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to match the existing new-client.py script. The old one accepts root-level proto_paths, so a PR with all the released versions get created. See this example PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was initially forcing the user to provide a versioned path until I saw the behavior of the old script

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have to follow the same behavior, its a tool that is used only by our own team, if the new behavior means less code and no additional toil, then we should go for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have to follow the same behavior, its a tool that is used only by our own team, if the new behavior means less code and no additional toil, then we should go for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. As discussed, I modified the script to support versioned proto_paths only.

For example, to generate the library for 'google/maps/routing/v2',
then you specify this value as 'google/maps/routing'
product_docs:
required: true
type: string
description: "`product_docs`: Documentation URL that appears in README.md"
rest_docs:
required: false
type: string
description: |
`rest_docs`: If it exists, link to the REST Documentation for a service
rpc_docs:
required: false
type: string
description: |
`rpc_docs`: If it exists, link to the RPC Documentation for a service
api_description:
diegomarquezp marked this conversation as resolved.
Show resolved Hide resolved
required: true
description: "`api_description`: Description that appears in README.md"
transport:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to specify transport. The value is parsed from BUILD.bazel.

required: false
type: choice
default: grpc
options:
- grpc
- http
- both
description: "`transport`: A label that appears in repo-metadata.json"
destination_name:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have a destination_name in configuration. Does this mean library_name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

library_name better resembles the new scripts, thanks. I'll update it

required: false
type: string
description: |
`destination_name`: The directory name of the new library. By default it's
java-<api_shortname>
distribution_name:
required: false
type: string
description: |
`distribution_name`: Maven coordinates of the generated library. By default it's
com.google.cloud:google-cloud-<api_shortname>

jobs:
generate:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.9'
cache: 'pip' # caching pip dependencies
- name: Install new-client.py dependencies
run: pip install --require-hashes -r generation/new_client_hermetic_build/requirements.in
- name: Add entry to generation_config.yaml
diegomarquezp marked this conversation as resolved.
Show resolved Hide resolved
id: config_generation
run: |
set -x
arguments="--api_shortname=\"${API_SHORTNAME}\" \
--proto-path=\"${PROTO_PATH}\" \
--name-pretty=\"${NAME_PRETTY}\" \
--product-docs=\"${PRODUCT_DOCS}\" \
--api-description=\"${API_DESCRIPTION}\""

# helper function that appends a python argument only if specified in the GH action inputs
append_argument() {
py_arg=$1
# env vars look exactly like new-client arguments but uppercase + underscores
env_name=$(echo "${py_arg}" | sed 's/-/_/g' | sed -e 's/\([a-z]\)/\U\1/g')
if [[ -n "${!env_name}" ]]; then
# $(echo) is redundant but it works around a syntax highlighting problem in vim
arguments=$(echo "${arguments} --${py_arg}=\"${!env_name}\"")
fi
}

declare -a optional_args=('transport' 'destination-name' 'distribution-name' 'group-id' 'rest-docs' 'rpc-docs')

for python_argument in "${optional_args[@]}"; do
append_argument "${python_argument}"
done
echo "::set-output name=new_library_args::${arguments}"
echo "${arguments} --googleapis-gen-url=\"${GOOGLEAPIS_GEN_URL}\"" \
| xargs python generation/new_client_hermetic_build/new-client.py generate
env:
GOOGLEAPIS_GEN_URL: https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_TOKEN }}@github.com/googleapis/googleapis-gen.git
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
NAME_PRETTY: ${{ github.event.inputs.name_pretty }}
PROTO_PATH: ${{ github.event.inputs.proto_path }}
PRODUCT_DOCS: ${{ github.event.inputs.product_docs }}
REST_DOCS: ${{ github.event.inputs.rest_docs }}
RPC_DOCS: ${{ github.event.inputs.rpc_docs }}
API_DESCRIPTION: ${{ github.event.inputs.api_description }}
TRANSPORT: ${{ github.event.inputs.transport }}
DESTINATION_NAME: ${{ github.event.inputs.destination_name }}
DISTRIBUTION_NAME: ${{ github.event.inputs.distribution_name }}
- name: setup docker environment
shell: bash
run: |
# we create a volume pointing to `pwd` (google-cloud-java) that will
# be referenced by the container and its children
if [[ $(docker volume inspect repo-google-cloud-java) != '[]' ]]; then
docker volume rm repo-google-cloud-java
fi
docker volume create --name "repo-google-cloud-java" --opt "type=none" --opt "device=$(pwd)" --opt "o=bind"
- name: generate from configuration
shell: bash
run: |
repo_volumes="-v repo-google-cloud-java:/workspace/google-cloud-java"
docker run --rm \
${repo_volumes} \
-v /tmp:/tmp \
-v /var/run/docker.sock:/var/run/docker.sock \
-e "RUNNING_IN_DOCKER=true" \
-e "REPO_BINDING_VOLUMES=${repo_volumes}" \
gcr.io/cloud-devrel-public-resources/java-library-generation:latest \
python /src/generate_repo.py generate \
--generation-config-yaml=/workspace/google-cloud-java/generation_config.yaml \
--repository-path=/workspace/google-cloud-java \
--target-library-api-shortname=${API_SHORTNAME}
env:
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
- name: Push to branch and create PR
run: |
set -x
[ -z "`git config user.email`" ] && git config --global user.email "cloud-java-bot@google.com"
[ -z "`git config user.name`" ] && git config --global user.name "cloud-java-bot"

# create and push to branch in origin
# random_id allows multiple runs of this workflow
random_id=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 5; echo)
branch_name="new-library/${{ github.event.inputs.api_shortname }}-${random_id}"
git checkout -b "${branch_name}"
git add --all
commit_message="feat: [${API_SHORTNAME}] new module for ${API_SHORTNAME}"
git commit -m "${commit_message}"
git remote add monorepo https://cloud-java-bot:${GH_TOKEN}@github.com/${{ github.repository }}.git
git fetch -q --unshallow monorepo
git push -f monorepo "${branch_name}"

# create PR
pr_body="Generated by @${USERNAME} via [generate_new_client_hermetic_build.yaml](https://github.com/googleapis/google-cloud-java/actions/workflows/generate_new_client_hermetic_build.yaml)

Command used:

\`\`\`
python generation/new_client_hermetic_build/new-client.py generate ${GENERATION_ARGUMENTS}
\`\`\`"
gh pr create --title "${commit_message}" --label "owlbot:run" --head "${branch_name}" --body "${pr_body}"
env:
USERNAME: ${{ github.actor }}
API_SHORTNAME: ${{ github.event.inputs.api_shortname }}
GENERATION_ARGUMENTS: ${{ steps.config_generation.outputs.new_library_args }}
GH_TOKEN: ${{ secrets.CLOUD_JAVA_BOT_TOKEN }}

Loading