dev: remove nginx

[keegancsmith]

We already have Caddy running for HTTPS support. We can use just Caddy
to remove the need to have nginx in our dev environments. This does mean
we won't be using the same reverse proxy as we do in production and for
customers. However, I do not think that changes much in practice.

This was inspired by a conversation I had with @mrnugget. Not completely sold on the idea though after implementing the change (especially since we have caddy -> webpack -> caddy -> frontend now).

@felixfbecker is this still an issue for you? https://github.com/sourcegraph/sourcegraph/pull/10021

[slimsag]

This does mean we won't be using the same reverse proxy as we do in production and for customers.

Production/customer environments run various reverse proxies:

• Managed instances: Google Cloud Load Balancer
• Kubernetes: k8s ingress OR nginx-svc
• Docker Compose: Caddy
• Pure Docker: Apache
• Server: nginx(?)

[slimsag]

I didn't review but signaling my approval for this idea.

[mrnugget]

I'm all for it. Less processes, less config files and while the reverse-reverse-proxy-stuff is not the prettiest, I like how you documented it now — much clearer than what we had before.

[felixfbecker]

We already have Caddy running for HTTPS support.

Caddy with HTTPS still doesn't work for me and using an extra workaround nginx config file with HTTPS config is still the only way I can get a functioning dev environment. See https://github.com/sourcegraph/sourcegraph/issues/10103. Unfortunately I have zero experience with Caddy (as opposed to nginx) and because I get no error messages from Caddy I have no idea how to solve this by myself. How can I keep a functioning dev environment after this PR?

[keegancsmith]

@felixfbecker can you try out this branch? We bump the caddy version, which might help resolve the issue.

[mrnugget]

@felixfbecker Could you try this again? It would be cool if we can get rid of one of two services.

[codecov]

Codecov Report

Merging #13299 (374674b) into main (dbd62f6) will increase coverage by 1.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #13299      +/-   ##
==========================================
+ Coverage   50.94%   52.06%   +1.12%     
==========================================
  Files        1749     1506     -243     
  Lines       87687    83520    -4167     
  Branches     7964     6977     -987     
==========================================
- Hits        44671    43488    -1183     
+ Misses      39079    36435    -2644     
+ Partials     3937     3597     -340     

Flag	Coverage Δ
go	52.70% <ø> (+2.91%)	⬆️
integration	29.54% <ø> (-0.58%)	⬇️
storybook	15.77% <ø> (-14.73%)	⬇️
typescript	50.34% <ø> (-3.36%)	⬇️
unit	34.10% <ø> (-1.40%)	⬇️

Impacted Files	Coverage Δ
...cker-images/prometheus/cmd/prom-wrapper/silence.go	0.00% <0.00%> (-90.00%)	⬇️
...end/graphqlbackend/repository_text_search_index.go	0.00% <0.00%> (-38.58%)	⬇️
...ocker-images/prometheus/cmd/prom-wrapper/status.go	0.00% <0.00%> (-34.32%)	⬇️
...d/frontend/graphqlbackend/externallink/resolver.go	0.00% <0.00%> (-33.34%)	⬇️
cmd/frontend/backend/repos_vcs.go	21.05% <0.00%> (-32.80%)	⬇️
enterprise/internal/codeintel/gitserver/client.go	0.00% <0.00%> (-27.90%)	⬇️
...er-images/prometheus/cmd/prom-wrapper/receivers.go	50.00% <0.00%> (-17.71%)	⬇️
cmd/frontend/graphqlbackend/repository_mirror.go	10.60% <0.00%> (-14.16%)	⬇️
cmd/frontend/internal/app/debug.go	17.14% <0.00%> (-13.90%)	⬇️
cmd/frontend/graphqlbackend/search_symbols.go	11.83% <0.00%> (-12.32%)	⬇️
... and 2749 more

[sourcegraph-bot]

Notifying subscribers in CODENOTIFY files for diff 58707f3...2cf5028.

Notify	File(s)
@christinaforney	doc/dev/getting-started/quickstart_1_install_dependencies.md doc/dev/how-to/troubleshooting_local_development.md

[mrnugget]

Works for me!

[felixfbecker]

This is still not working for me, however now I at least seem to get an error message. I have no idea how to resolve this though, Caddy feels like a blackbox to me 😬

11:03:13                     caddy | {"level":"error","ts":1613556193.239407,"logger":"tls.renew","msg":"will retry","error":"open /Users/felix/Library/Application Support/Caddy/certificates/local/sourcegraph.test/sourcegraph.test.crt: no such file or directory","attempt":2,"retrying_in":120,"elapsed":60.004151059,"max_duration":2592000}

[mrnugget]

Does the rest of the path exist? /Users/felix/Library/Application Support/Caddy/certificates/local/sourcegraph.test?

What happens if you delete the whole folder, /Users/felix/Library/Application Support/Caddy, and start with ./dev/caddy.sh trust and then ./dev/caddy.sh run --watch --config=dev/Caddyfile?

(btw. side note: if you're not using Caddy, how do you use HTTPS?)

[keegancsmith]

@felixfbecker also see our troubleshooting guide https://github.com/sourcegraph/sourcegraph/blob/97c9b9b62a1ad319e437f17c28b891ebac3f20b0/doc/dev/how-to/troubleshooting_local_development.md#caddy-2-certificate-problems

[eseliger]

seems to work on my machine, so LGTM
nice job cleaning up this mess a bit 😁

[felixfbecker]

@mrnugget's steps worked for me. Thanks!

[felixfbecker]

(btw. side note: if you're not using Caddy, how do you use HTTPS?)

I had a custom gitignored nginx config with a certificate generated by mkcert. I added support for sourcing custom nginx configs right when we switched to Caddy as a workaround because Caddy was just not working for me (meaning my dev env was broken and I needed an immediate solution) and nobody could help me 🤷‍♂️. I'm certainly happy to be able to get rid of that!

[mrnugget]

Sweet! Happy to hear that!