add page for repo data on dot com, how to add repos to index

[rvantonder]

There's a recurring use case/request for how to add custom repositories to index on Sourcegraph.com. It's not documented. This PR is a best effort to document how to do it currently.

Please check the commands for accuracy @daxmc99 (is it still OK to use kubectl instead of gcloud? is there an alternative gcloud command?)

Please check that this page lives in a sensible place @sourcegraph/distribution

[sourcegraph-bot]

Notifying subscribers in CODENOTIFY files for diff 496118f...e975cb6.

Notify	File(s)
@nicksnyder	handbook/engineering/deployments.md handbook/engineering/distribution/repository_data_on_sourcegraph_dot_com.md
@sourcegraph/distribution	handbook/engineering/distribution/repository_data_on_sourcegraph_dot_com.md

[daxmc99]

This will need to be changed to connect to the Cloud SQL database.
The proper command will gcloud beta sql connect --project sourcegraph-dev sg-cloud-732a936743 --user=sg -d=sg
The password is available in our shared 1password under Google Cloud SQL - default db

The beta here requires the cloud SQL proxy which is preferred since otherwise, it requires a temp whitelist of your IP in order to access Cloud SQL.

[rvantonder]

@daxmc99 got it, what's the right page to link to for setting up gcloud?

Also, the kubectl command appears to still work, will that just break at some point?

[daxmc99]

Yes, the pgsql pod is currently only there to provide a comparison to ensure that we didn't lose any performance. That command will stop working once that pod is removed.

The best way to setup the gcloud cli would be here https://cloud.google.com/sdk/docs/install

[poojaj-tech]

@rvantonder I ran into this access related error. Suggesting this addition for others who might run into this too.

Error

psql: error: could not connect to server: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

This can happen if you don't have access to the project you are trying to connect to. Contact the distribution team to get access.

[keegancsmith]

cc @ryanslade @asdine

[tsenart]

I think it'd be best to actually dog food the work we've been doing to support user added repos in @sourcegraph/cloud instead of documenting direct database insertion into the default_repos table. This should be possible at the end of this week I believe. Please confirm @asdine @ryanslade.

[ryanslade]

Agreed. Our work should be ready for dogfooding within the next day or so.

[asdine]

I have just merged sourcegraph/sourcegraph#14341, all user-added external services will be indexed automatically alongside those already present in the default_repos table. We can start adding non admin external services and expect them to be indexed.

[tsenart]

We can start adding non admin external services and expect them to be indexed.

We need to make a demo video to show teammates how this is done.

[asdine]

We need to make a demo video to show teammates how this is done.

Or perhaps that's a good subject for a Brain Food lightning talk? Since we now have an internal platform for presenting stuff to our teammates I think we should leverage it

[tsenart]

Or perhaps that's a good subject for a Brain Food lightning talk? Since we now have an internal platform for presenting stuff to our teammates I think we should leverage it

That's complementary — a demo video allows for easier sharing. We want to show our progress and be able to potentially publish these in release blog posts.

[tsenart]

I filed sourcegraph/sourcegraph#14438. In the meantime, do we need to enable any specific feature flags for all teammates to see the UI?

[rvantonder]

I think it'd be best to actually dog food the work we've been doing to support user added repos in @sourcegraph/cloud instead of documenting direct database insertion into the default_repos table. This should be possible at the end of this week I believe. Please confirm @asdine @ryanslade.

Since this sounds confirmed, please open another PR for the handbook for this and I'll close this one? Video is nice but a very "online" mode of communicating something, there should be some textual description of how we index repo data :-)

[tsenart]

Video is nice but a very "online" mode of communicating something, there should be some textual description of how we index repo data :-)

Agreed. Filed sourcegraph/sourcegraph#14477