Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(runjob): add support for cloud_sql_instance #1688

Conversation

CyberHippo
Copy link
Contributor

Change description

Add support for cloud_sql_instance for Cloud Run Jobs.

Fixes #1307

Tests you have done

  • Run make ready-pr to ensure this PR is ready for review.
  • Perform necessary E2E testing for changed resources.
  • go test -v -tags=integration ./config/tests/samples/create -test.run TestAll -run-tests runjob

@CyberHippo CyberHippo changed the title feat(runjob): add sqlinstance field feat(runjob): add support for cloud_sql_instance May 3, 2024
@CyberHippo
Copy link
Contributor Author

/test all

Copy link
Contributor

@CyberHippo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@CyberHippo
Copy link
Contributor Author

Hi, @acpana / @anhdle-sso / @barney-s / @cheftako / @diviner524 / @gemmahou / @jasonvigil / @jiefenghe / @jingyih / @justinsb / @maqiuyujoyce / @roycaihw / @yuwenma / @ziyue-101 /@xiaoweim sorry for tagging you directly, could you please take a look at my PR. This would greatly improve the usability of RunJob for us. Thanks !

@gemmahou
Copy link
Collaborator

gemmahou commented Jul 1, 2024

/ok-to-test

Copy link
Contributor

@gemmahou: No presubmit jobs available for GoogleCloudPlatform/k8s-config-connector@master

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gemmahou
Copy link
Collaborator

gemmahou commented Jul 1, 2024

/test all

Copy link
Contributor

@gemmahou: No presubmit jobs available for GoogleCloudPlatform/k8s-config-connector@master

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gemmahou
Copy link
Collaborator

gemmahou commented Jul 1, 2024

Hi @CyberHippo would you mind collaborating more on your issue #1307? Could you help us understand the business impact and the priority to support this feature? Is there a KCC/Google internal feature request associate to it? Thanks!

@CyberHippo
Copy link
Contributor Author

Hi @gemmahou , thank you for taking a look!

would you mind collaborating more on your issue #1307?

Sure, let me know what information you need.

Could you help us understand the business impact and the priority to support this feature?

We are deploying compute heavy workloads using Cloud Run Jobs and these workloads need to access our Cloud SQL instances. As of now we deploy these Run Jobs using KCC but the lack of Cloud SQL instance support means that we need to create and use Cloud SQL client certificates instead of relying on the SQL socket. It seems that Cloud Run Jobs already support this feature and that it only needs to be added in the RunJob specs for it to work. Please note that KCC Run Service supports Cloud SQL instances.

Is there a KCC/Google internal feature request associate to it?

Not that I am aware of but I could open one if needed. I'm not familiar with KCC dev cycle.

@CyberHippo
Copy link
Contributor Author

Friendly ping @gemmahou

@gemmahou
Copy link
Collaborator

Friendly ping @gemmahou

Here's the guide for asking a new feature support: https://g3doc.corp.google.com/cloud/config/configconnector/g3doc/customer-engagement.md#recommended-support-channels

We suggest open a GH or Buganizier issue, describe the stakeholder, business impact, and assign to KCC team member or our PM Nic(nslattery). That will help us manage and track feature requests.

cc: @yuwenma

- tfField: template.template.volumes.cloud_sql_instance.instances
description: |-
The Cloud SQL instance connection names, as can be found in https://console.cloud.google.com/sql/instances. Visit https://cloud.google.com/sql/docs/mysql/connect-run for more information on how to connect Cloud SQL and Cloud Run. Format: {project}:{location}:{instance}
key: instances
Copy link
Collaborator

@gemmahou gemmahou Jul 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be instancesRef per the naming convention: https://github.com/GoogleCloudPlatform/k8s-config-connector/blob/master/README.ConfigureResourceReferences.md#configure-reference-resource-in-the-service-mappings, point 3

Edit: Please ignore this comment. Since this is a List field, currently we use fieldName directly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for a list-typed reference field, we should use name instanceRefs instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I used instances to be coherent with the instance field of Run Services but I changed it to instanceRefs in 249d217

@gemmahou
Copy link
Collaborator

gemmahou commented Jul 11, 2024

We require test coverage for new field reference: https://github.com/GoogleCloudPlatform/k8s-config-connector/blob/master/README.NewResourceFromTerraform.md#test-your-changes

Here are some existing test cases for your reference:
Dynamic test: https://github.com/GoogleCloudPlatform/k8s-config-connector/tree/master/pkg/test/resourcefixture/testdata/basic/run/v1beta1/runjob
Samples test: https://github.com/GoogleCloudPlatform/k8s-config-connector/tree/master/config/samples/resources/runjob

None of above tests covered the CloudSQLInstance added in this PR, could you add a new test case(i.e. jobwithsqlinstance), include the field reference(CloudSQLInstance) you add, run the dynamic and samples test to make sure it works as expected?

@CyberHippo
Copy link
Contributor Author

/test all

Copy link
Contributor

@CyberHippo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@CyberHippo
Copy link
Contributor Author

Hi @gemmahou , as suggested I added dynamic tests for Run job with SQL deps (please see f13a8de).

@gemmahou
Copy link
Collaborator

gemmahou commented Aug 1, 2024

@CyberHippo triggered the presubmit test. Could you let us know who is the stakeholder/customer for this feature request? Thanks!

@CyberHippo
Copy link
Contributor Author

@CyberHippo triggered the presubmit test. Could you let us know who is the stakeholder/customer for this feature request? Thanks!

Hi @gemmahou, thanks for running the presubmit job ! The customer would be the company elmy.

Copy link
Collaborator

@gemmahou gemmahou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for adding the test! Only one comment regarding the field name.

for more information on how to connect Cloud SQL and Cloud
Run.
properties:
instances:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC this file is generated and field name should be updated to instanceRefs as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I missed it running make manifests, fixed with 26dbc18

@gemmahou
Copy link
Collaborator

gemmahou commented Aug 6, 2024

/lgtm
/approve

@google-oss-prow google-oss-prow bot added the lgtm label Aug 6, 2024
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gemmahou

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 03a3d7d into GoogleCloudPlatform:master Aug 6, 2024
12 of 13 checks passed
@CyberHippo CyberHippo deleted the feat/run-job-sql-instances branch August 8, 2024 06:28
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to configure Cloud SQL instance for a RunJob ?
3 participants