Clarify steps needed for Cloud SQL #1928

daxmc99 · 2020-11-02T22:34:08Z

This adds additional clarification for how to access our production databases

Followup to https://sourcegraph.slack.com/archives/D012ZE3Q5J7/p1604332857000100

sourcegraph-bot · 2020-11-02T22:34:56Z

Notifying subscribers in CODENOTIFY files for diff b807166...6a24673.

Notify	File(s)
@nicksnyder	handbook/engineering/deployments/postgresql.md

nicksnyder · 2020-11-02T23:28:47Z

handbook/engineering/deployments/postgresql.md

+
+- Default db {[Password](https://my.1password.com/vaults/dnrhbauihkhjs5ag6vszsme45a/allitems/svfiw4vcbxhhbobpl442olyebu/)}
+  ```
+    gcloud beta sql connect --project sourcegraph-dev sg-cloud-732a936743 --user=sg -d=sg


Doesn't work for me

% gcloud beta sql connect --project sourcegraph-dev sg-cloud-732a936743 --user=sg -d=sg ERROR: (gcloud.beta.sql.connect) Cloud SQL Proxy could not be found in PATH. See https://cloud.google.com/sql/docs/mysql/sql-proxy#install for information on installing.

Ahh, I thought it prompted to install the beta utils if you did not have them

You do not currently have this command group installed. Using it requires the installation of components: [beta] Your current Cloud SDK version is: 310.0.0 Installing components from version: 310.0.0 ┌─────────────────────────────────────────────┐ │ These components will be installed. │ ├──────────────────────┬────────────┬─────────┤ │ Name │ Version │ Size │ ├──────────────────────┼────────────┼─────────┤ │ gcloud Beta Commands │ 2019.05.17 │ < 1 MiB │ └──────────────────────┴────────────┴─────────┘ For the latest full release notes, please visit: https://cloud.google.com/sdk/release_notes Do you want to continue (Y/n)?

What makes you think I don't have beta tools installed?

It seems like the cloud-sql-proxy is not installed as part of the beta tools. I thought it was.

nicksnyder · 2020-11-02T23:28:56Z

handbook/engineering/deployments/postgresql.md

+
+You may use these gcloud commands to connect directly to the databases:
+
+- Default db {[Password](https://my.1password.com/vaults/dnrhbauihkhjs5ag6vszsme45a/allitems/svfiw4vcbxhhbobpl442olyebu/)}


how/where is the password provided?

The login should prompt for it

nicksnyder · 2020-11-02T23:29:20Z

handbook/engineering/deployments/postgresql.md

+https://console.cloud.google.com/bigquery?sq=527047051561:67f2616f4acb4b7cb3639e4a97e2f4aa
+) and their `EXTERNAL_QUERY` syntax.
+
+There are two ways of connecting: either using the `gcloud beta sql connect` command or running the proxy on a port locally.


As a developer, how do I know which way to choose?

gcloud is mainly for quick pgsql commands, the proxy is used when you want to connect an external database viewer or GUI program.

Suggested change

There are two ways of connecting: either using the `gcloud beta sql connect` command or running the proxy on a port locally.

There are two ways of connecting: either using the `gcloud beta sql connect` command which will use the `pgsql` client or running the proxy on a port locally to utilize whatever tools you typically use.

nicksnyder · 2020-11-02T23:30:17Z

handbook/engineering/deployments/postgresql.md

+  ```
+  _Note:_ You may need to set or unset your PGPASSWORD environment variable to the value of the password of the user you are using to connect (`-U sg`).
+
+If you are frequently using this to access the database for read-only operations, please let the #dev-ops channel know so we can add a read-only replica.


At the very least, it seems like we should create a readonly user.

Added here https://github.com/sourcegraph/infrastructure/pull/2234

bobheadxi · 2020-11-10T00:38:00Z

handbook/engineering/deployments/postgresql.md

+For read-only access, there is also an option of using [BigQuery](
+https://console.cloud.google.com/bigquery?sq=527047051561:67f2616f4acb4b7cb3639e4a97e2f4aa
+) and their `EXTERNAL_QUERY` syntax.


I think using BigQuery might deserve a section of its own, with the full EXTERNAL_QUERY here (did not realize that it was just a stand-in for a table), e.g. seeing:

SELECT * FROM EXTERNAL_QUERY("...", "SELECT name::text,created_at::text FROM repo LIMIT 5;");

would be a lot more self-explanatory. Seems a lot less setup than the alternative, so for UI-and-mouse noobs like me I would probably go for this more often than not 😛

handbook/engineering/deployments/postgresql.md

Co-authored-by: Robert Lin <robert@bobheadxi.dev>

nicksnyder · 2020-11-10T19:21:08Z

don't wait on my review, ultimately this documentation should make it clear to a new teammate how to connect to the database. The best review here is to have a new teammate walk through these steps and give you feedback on whether they were successful or not (feel free to merge first)

flying-robot · 2020-11-11T14:15:50Z

handbook/engineering/deployments/postgresql.md

+
+- Default db {[Password](https://start.1password.com/open/i?a=HEDEDSLHPBFGRBTKAKJWE23XX4&v=dnrhbauihkhjs5ag6vszsme45a&i=pjxf64qxwsin4d56xij6vm3gva&h=my.1password.com)}
+  ```
+    gcloud beta sql connect --project sourcegraph-dev sg-cloud-732a936743 --user=dev-readonly -d=sg


I had to export CLOUDSDK_PYTHON=python2. I have both 2 and 3 installed, 2 appears first in my path, but it looks like 3 gets chosen anyway:

Python 3 is preferred over Python 2. Note that gcloud requires Python version 2.7.x or 3.5-3.8.

I have 2.7 and 3.9, so without the environment variable I get AttributeError: module 'importlib' has no attribute 'util'.

Do you find this command useful or do you just use the proxy by itself?

For me personally, the command is useful. I don't generally use a GUI or similar tool for PG access.

handbook/engineering/deployments/postgresql.md

flying-robot

Worked for me, just left a couple of notes/questions

Clarify steps needed for Cloud SQL

42b332b

daxmc99 requested review from nicksnyder and pecigonzalo November 2, 2020 22:34

Highlight two distinct choices

f05cd01

nicksnyder reviewed Nov 2, 2020

View reviewed changes

The cloud sql proxy must be installed manually

b5344d4

daxmc99 marked this pull request as draft November 6, 2020 18:58

daxmc99 added 2 commits November 6, 2020 18:52

Switch to using dev users instead of "sg" users

699056b

grammar, readonly user

2589238

daxmc99 marked this pull request as ready for review November 7, 2020 01:58

daxmc99 requested a review from nicksnyder November 9, 2020 17:43

bobheadxi reviewed Nov 10, 2020

View reviewed changes

handbook/engineering/deployments/postgresql.md Outdated Show resolved Hide resolved

Update handbook/engineering/deployments/postgresql.md

bb8150e

Co-authored-by: Robert Lin <robert@bobheadxi.dev>

pecigonzalo added this to the Dist: 2020.11.03 milestone Nov 10, 2020

pecigonzalo added the team/distribution label Nov 10, 2020

sourcegraph-bot mentioned this pull request Nov 10, 2020

Distribution: 2020.11.03 - Amazing Otter sourcegraph/sourcegraph#14630

Closed

52 tasks

flying-robot reviewed Nov 11, 2020

View reviewed changes

handbook/engineering/deployments/postgresql.md Outdated Show resolved Hide resolved

flying-robot approved these changes Nov 11, 2020

View reviewed changes

Update handbook/engineering/deployments/postgresql.md

6a24673

daxmc99 merged commit b505b7b into main Nov 12, 2020

daxmc99 deleted the dax/clarify_postgres_connection branch November 12, 2020 06:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clarify steps needed for Cloud SQL #1928

Clarify steps needed for Cloud SQL #1928

daxmc99 commented Nov 2, 2020 •

edited

Loading

sourcegraph-bot commented Nov 2, 2020 •

edited

Loading

nicksnyder Nov 2, 2020

daxmc99 Nov 2, 2020

nicksnyder Nov 2, 2020

daxmc99 Nov 2, 2020

nicksnyder Nov 2, 2020

daxmc99 Nov 6, 2020

nicksnyder Nov 2, 2020

daxmc99 Nov 2, 2020

daxmc99 Nov 2, 2020

nicksnyder Nov 2, 2020

daxmc99 Nov 6, 2020

bobheadxi Nov 10, 2020

nicksnyder commented Nov 10, 2020

flying-robot Nov 11, 2020

daxmc99 Nov 11, 2020

flying-robot Nov 11, 2020

flying-robot left a comment


		You may use these gcloud commands to connect directly to the databases:

		- Default db {[Password](https://my.1password.com/vaults/dnrhbauihkhjs5ag6vszsme45a/allitems/svfiw4vcbxhhbobpl442olyebu/)}

	There are two ways of connecting: either using the `gcloud beta sql connect` command or running the proxy on a port locally.
	There are two ways of connecting: either using the `gcloud beta sql connect` command which will use the `pgsql` client or running the proxy on a port locally to utilize whatever tools you typically use.

Clarify steps needed for Cloud SQL #1928

Clarify steps needed for Cloud SQL #1928

Conversation

daxmc99 commented Nov 2, 2020 • edited Loading

sourcegraph-bot commented Nov 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksnyder commented Nov 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flying-robot left a comment

Choose a reason for hiding this comment

daxmc99 commented Nov 2, 2020 •

edited

Loading

sourcegraph-bot commented Nov 2, 2020 •

edited

Loading