-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: define HTTP authentication extension mechanism #26232
Comments
The original vgo prototype just read $HOME/.netrc. I'd rather do something like that than shell out to a whole separate program. |
.netrc works for HTTP basic auth with unlimited lifetime credentials. But it doesn't work for other types, like:
|
Note that even "access denied" pages can have meta tags - go get does not require a 200 response, exactly for this kind of thing. So git.mycompany.com could serve the tags unauthenticated as one workaround. Are there any more general "HTTP auth credential helpers" in the world besides git-credential-helper? How do other systems deal with this? If there's something standard to hook into (like .netrc) then that's better than designing our own. |
If the access denied page contains the meta tag, it can be used as a prober. Let's say git.mycompany.com has an not-yet announced new product, collaborating with another company other-tech-company.com, and they host a Git repository at git.mycompany.com/ng-product/collab-other-tech-company. An attacker can probe if a repository exists by looking at the go-get's meta tag, and they can learn mycompany.com will have a business relationship other-tech-company.com to create a new product (and maybe benefit from this insider info). As far as I know, there's no generic mechanism that majority of people use for storing and obtaining credentials through API. The closest thing I can think of is ssh-agent. Or if it's OS X, OS X Keychain. |
It's only a prober if you check that the thing exists before serving the meta-tag.
|
Clearly we could use a better idea here but I'd like to have just one thing that's standard instead of inventing our own (that is, I want something like .netrc). |
I cannot interpret this. The problem is that an attacker can use this to extract the repository existence by using this meta-tag. If the page checks if the repository exists and change the response, it can be used as a prober. |
Then don't check, and don't change the response. |
https://docs.docker.com/engine/reference/commandline/login/#credentials-store https://github.com/GoogleCloudPlatform/docker-credential-gcr |
So the page should not do any auth, and should not contain meta tag? I thought you're suggesting to add a meta tag and in the next response you're saying not to add meta tag? |
An example leakage: https://play.golang.org/p/rCYzVWQSQni We want to avoid this. By adding a meta-tag to the response as rsc suggests, the repository existence is leaked. |
@draftcode, I assumed what was meant as you should advertise it for all URLs, including /android/device/g000gle/foo-bar-not-exist , regardless of whether it actually exists. |
Note that the third part in meta tag should be a git repository URL. This means, if a repository exists, the third part should be a valid Git repository path. In the example, I used |
An example that for this: https://play.golang.org/p/OWb6zRUD02r |
@draftcode I am saying that if you pick a trivial rule like "example.com/x/y/z/w redirects to the repo at git.example.com/x/y", then you can implement it with no Auth check and no repo validity check, so that it responds to any possible x/y/z/w with the same answer. Then there is no leakage. That was the point of my curl example above: there is no actual package at rsc.io/asdgihopajdfklgadfglhifsdfj but the curl still reads back meta tags for it. If you need example.com/x/y/z/w to redirect to different git servers based on the exact values of x,y,z,w then that's more difficult of course and leakage is hard to avoid. This is all just a suggested workaround. What I'd really like is to find out about some plausibly general (not git-specific, not docker-specific) semi-standard way to interact with credential helpers. Maybe none exists and we have to come up with something. But that would be my last choice. |
OK. We've considered that option when we investigated how GitHub deals with this, and found out that GitHub won't allow slashes in a repository name, so they can do what you've said. We cannot do that, so filed this bug.
So far, I've shown what Git does for a similar problem. @bradfitz showed what Docker and GCP does for a similar problem (ADC now works only for service accounts, so it's a bit different). If there's a some standard way to get a cred, considering the size of these tools' community, there should be some implementation of that, but it seems there's no such thing. In fact, Docker created a credential helper mechanism following what Git did. From these, such standard, if exists, is something that Git and Docker communities at least are not aware of and are not using. @rsc, @bradfitz, and I are also not aware of such generic way that would be called as "semi-standard", it seems. |
Given that there doesn't seem to be an agreed-upon standard, I guess the next step is for someone to propose a design that Go should use. I looked at git credential helpers but stopped looking when I saw bash snippets. |
Questions on the requirements:
|
He objected to bash snippets. It is possible (I do not know) that executing a binary rather than a shell might sit better with him. If it were indeed command execution (a la |
Command-line execution seems necessary, since you want to lock things down and give cmd/go access to just one password, not your whole password set. Josh, what did you have in mind? Want to propose a starting point design? |
It would be great if we could achieve the stated above goal ("Make Assuming the server returns a meta tag such as
I believe this would make possible to use git with the SSH auth without having everyone to add the I think it is an incredible strength of the Go tool that you can install and use it "right out of the box" - would love to see that feature and stellar user experience preserved and extended! 😄 [1] https://go-review.googlesource.com/c/go/+/33171 (abandoned; issue https://golang.org/issue/17898) |
@dmitris This proposal is for the servers that cannot return meta tags without an auth (see #26232 (comment)). It seems to me that your problem is not related to this. |
Just bumped into this while trying to get at a gitlab instance that was behind a cloudflare auth. So commenting here to add to the use-cases. I'd like to be able to set, somehow, a custom http header to allow cloudflare access e.g.
If I could configure this custom header, in a .netrc style, so that I can just say that matching hosts have to supply the extra header value, it would have solved the access problem early nicely |
Client TLS cert auth to private git repos would be nice also: #53197 |
Seems like there are a few ways to do authentication, one of them using the header to send a token. I am also facing the same issue as @srowles . As such, would it be useful to split this task into multiple smaller ones? One of them to add the reading of a header and its corresponding value from Go's environment variable (i.e. GOAUTH_HEADER, GOAUTH_HEADER_VALUE)? |
Looks like GCP are trying to do stuff with artefact registry now and go module proxy. So looks like there is a need to support something |
Creates the GOAUTH env var, and allows for credential helper invocation to provide arbitrary credentials for module servers. updates golang#26232
I am very late to this party it seems, but I might have a bit more to offer than just a +1. netrc as the only supported authentication mechanism for module proxies doesn't seem to be a passionate issue anywhere, but like @owenhaynes pointed out, module adoption is widespread and having to manage git/ssh credentials for private modules is a growing thorn. I'll go as far as to say I think programatically updating .netrc is probably more a sign of desperate madness than a robust solution. For better or worse, I didn't find this issue until after I'd taken a stab at fixing it myself. Fortunately, I came to most of the same conclusions this proposal did, with a few exceptions. I agree entirely with avoiding bespoke protocols where good ones exist, but also that existing protocols that leverage plaintext and hardcoded tokens are best left behind. The few assumptions I think I might have made differently to @bcmills:
My design is as follows:The The grammar for GOAUTH='*priv.example.com!/usr/local/bin/corp-credhelper;--user=me,proxy.example.com/private/*!my-credtool' In this, The protocol between go and the helper is borrowed from a system that seems to be working well in the real world: kubernetes and its
The Test codeI have this implementation "working" in a dev branch: master...AndrewBurian:go:auth-plugins This is my first time seriously slogging my way through go's test suite, so forgive... everything. It's mostly a PoC so I could see if the design was even a good idea or not, and seems to work as I'd expect. Git+SSH?I'm not sure if this helps or hurts the situation with Github and ssh auth #26134. My understanding after catching up all the discussion here is that go defers everything about the transport of source code outside of module zips to the VSC tool, and so having to configure git separately with auth for modules being served from private VCS feels correct. The focus of There may be some weird behaviour if the http authentication allows go to discover the existence of a repo with the initial https I will continue to play around with this, but after finally getting it to all compile my go seamlessly fetches modules with |
maybe instead of implementing the myriad of authentication strategies people may want, we just have a making of module prefixes to plain http proxies |
The lack of the ability to set client certificates for the module resolver
to connect to modules behind mTLS repos/gitlab/etc is the reason I am
currently forced to use a proxy. Being able to set client side
certificates and certificates authorities would be a nice addition and
would eliminate the need for those like me who are forced to proxy around
the security that mTLS provides.
…On Wed, Mar 22, 2023 at 11:31 AM Sean Liao ***@***.***> wrote:
maybe instead of implementing the myriad of authentication strategies
people may want, we just have a making of module prefixes to plain http
proxies
and let the proxies be responsible for the auth
—
Reply to this email directly, view it on GitHub
<#26232 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPPCRM6N4ZR4FAE3NIODQ3W5MLNVANCNFSM4FIQKQQQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
The GOAUTH env var is not yet recognized right? At least on 1.20.4 it is being ignored |
Any news on this? I've seen that this has been worked on (https://github.com/golang/tools/tree/master/cmd/auth), but as @rsj-ioki says it does not seem that it is fully implemented or available. |
The lack of any authentification mechanism against a |
Whoever works on this, please make sure that pkgsite commands use the same mechanism. Currently, it is very difficult to get a private documentation server running, and even once it's up, the lack of even netrc support causes all source links to be broken. |
Weird behavior encountered on GitLab's subgroups in private mode. After investigation, it turns out that GitLab conceils the existence of the nested project when unauthenticated: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/middleware/go.rb#L114-126. The current behavior of the library is to rely on the `git ls-remote -q` command when the path contains a `.git`, which relies on gitcredentials for auth. The exact behavior is detailed in this comment: golang/go#26232 (comment) Interesting link regrouping all Go private repo issues: golang/go#29953 Signed-off-by: grouville <guillaume@dagger.io>
Weird behavior encountered on GitLab's subgroups in private mode. After investigation, it turns out that GitLab conceils the existence of the nested project when unauthenticated: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/middleware/go.rb#L114-126. The current behavior of the library is to rely on the `git ls-remote -q` command when the path contains a `.git`, which relies on gitcredentials for auth. The exact behavior is detailed in this comment: golang/go#26232 (comment) Interesting link regrouping all Go private repo issues: golang/go#29953 Signed-off-by: grouville <guillaume@dagger.io>
Weird behavior encountered on GitLab's subgroups in private mode. After investigation, it turns out that GitLab conceils the existence of the nested project when unauthenticated: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/middleware/go.rb#L114-126. The current behavior of the library is to rely on the `git ls-remote -q` command when the path contains a `.git`, which relies on gitcredentials for auth. The exact behavior is detailed in this comment: golang/go#26232 (comment) Interesting link regrouping all Go private repo issues: golang/go#29953 Signed-off-by: grouville <guillaume@dagger.io>
Change https://go.dev/cl/605256 mentions this issue: |
Change https://go.dev/cl/605275 mentions this issue: |
Change https://go.dev/cl/605298 mentions this issue: |
This change adds a new environment variable GOAUTH which takes a semicolon-separated list of commands to run for authentication during go-import resolution and HTTPS module mirror protocol interactions. This CL only supports netrc and off. Future CLs to follow will extend support to git and a custom authenticator command. For #26232 Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest Change-Id: I6cfa4c89fd27a7a4e7d25c8713d191dc82b7e28a Reviewed-on: https://go-review.googlesource.com/c/go/+/605256 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Matloob <matloob@golang.org> Reviewed-by: Alan Donovan <adonovan@google.com>
NOTE: The accepted proposal is #26232 (comment).
Problem
The custom import path mechanism (
?go-get=1
and meta tag) works for public URLs, but it doesn't work for auth required URLs. This is because when go-get fetches the URL with?go-get=1
, it uses net/http.DefaultClient, and it doesn't know the credentials it needs to access the URL. A user cannot rungo get
against private source code hosting service because of this.Goal
Make
go get git.mycompany.com/private-repo
work, where https://git.mycompany.com/private-repo requires authentication.Idea 1 (credential helper)
Introduce a credential helper mechanism like git-credential-helpers. A user specifies a path to a binary via
GOGET_CREDENTIAL_HELPER
andgo get
executes that with the import path as an argument. The credential helper binary returns HTTP headers (like "Authorization: Basic blah\n", andgo get
adds these headers when it tries to fetchgo-get=1
URLs.Idea 2 (go-get helper)
Introduce a custom source code fetching mechanism. When
go get
needs to fetch the source code for the import pathgit.mycompany.com/private-repo
, it looks for the binarygo-get-git.mycompany.com
based on the host name of the import path. When such binary exists in $PATH, it executes that binary with the import path and a destination in $GOPATH (for example,go-get-git.mycompany.com git.mycompany.com/private-repo $GOPATH/src/git.mycompany.com/private-repo
). The binary is responsible for fetching the source code to the specified $GOPATH location.go get
work with VCSs other than git/hg/svn/bzr.go get -f
orgo get -insecure
.The text was updated successfully, but these errors were encountered: