GoMod: go list -deps error: no matching files found #5560

mawl · 2022-07-14T13:09:02Z

Hey,

We have an issue with the GoMod Package Manager

10:35:45.499 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.managers.GoMod - Resolving GoMod dependencies for path 'go.mod' failed with: 
IOException: Running 'go list -deps -f {{with .Module}}{{.Path}} {{.Version}}{{end}} ./...' in '/builds/some-go-app' failed with exit code 1:
XXXXXX.go:10:12: pattern some.file: no matching files found

Content of XXXXXX.go

//go:generate cp ../../../some.file .
//go:embed some.file

some.file ist generated during build of another component in the project.

We don't want to run "go generate/go build" as a prequisite for ort. So, are there other ways to get this working?

Cheers,
Marco

The text was updated successfully, but these errors were encountered:

fviernau · 2022-07-14T13:14:51Z

@mawl would you be able to share a minimal example to reproduce this issue?
Also which go version have you been using? Have you run it in most recent ORT docker?

mawl · 2022-07-14T14:22:22Z

@fviernau here is a small example: gomod-embed-non-existing-file.zip - i'm no go pro, but I can build it by running:

compile:
  image: golang:latest
  stage: build
  script:
    - mkdir -p mybinaries
    - go generate hello-world.go
    - go build -o mybinaries hello-world.go
  artifacts:
    paths:
      - mybinaries

ort version is: https://github.com/oss-review-toolkit/ort/tree/6de32461bd9f9d20de8550c56af94839498b425a

fviernau · 2022-07-14T14:40:19Z

Slightly related: golang/go#49534

mawl · 2022-07-19T06:48:39Z

As a workaround I use following script before running oss review toolkit, which removes the go:embed directive from relevant files (Leaving // as a comment)

grep -rl --include \*.go --exclude-dir=vendor "go:embed" . | xargs sed -i 's/go\:embed//g'

The `-json` option allows specifying a comma separated list of property names to include in the output. The `go list` command has a mechanism for skipping certain processing steps which are not needed for computing the values corresponding to those specified property names, see [^1] and [^2]. Such a mechanism is not in place for the `-f` option. Migrate from `-f` to `-json` to benefit from above mechanism, removing potential sources of issues. It could be that this allows to drop the parameter `-buildvcs=false`, which is left for future investigation. The primary reason for switching to `-json` is to extend the mentioned mechanism in `go list` to conditionally skip computing embed files in an analog way which would fix [^4]. [^1]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/load/pkg.go#L2768-L2776 [^2]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/list/list.go#L605-L606 [^3]: golang/go#49534 (comment) [^4]: #5560 Signed-off-by: Frank Viernau <frank_viernau@epam.com>

The `-json` option allows specifying a comma separated list of property names to include in the output. The `go list` command has a mechanism for skipping certain processing steps which are not needed for computing the values corresponding to those specified property names, see [^1],[^2] and [^3]. Such a mechanism is not in place for the `-f` option. Migrate from `-f` to `-json` to benefit from above mechanism, removing potential sources of issues. It could be that this allows to drop the parameter `-buildvcs=false`, which is left for future investigation. The primary reason for switching to `-json` is to extend the mentioned mechanism [^4]] in `go list` to conditionally skip computing embed files in an analog way which would fix [^5]. [^1]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/load/pkg.go#L2768-L2776 [^2]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/list/list.go#L605-L606 [^3]: golang/go#29666 [^4]: golang/go#49534 (comment) [^5]: #5560 Signed-off-by: Frank Viernau <frank_viernau@epam.com>

The `-json` option allows specifying a comma separated list of property names to include in the output. The `go list` command has a mechanism for skipping certain processing steps which are not needed for computing the values corresponding to those specified property names, see [^1],[^2] and [^3]. Such a mechanism is not in place for the `-f` option. Migrate from `-f` to `-json` to benefit from above mechanism, removing potential sources of issues. It could be that this allows to drop the parameter `-buildvcs=false`, which is left for future investigation. The primary reason for switching to `-json` is to extend the mentioned mechanism [^4] in `go list` to conditionally skip computing embed files in an analog way which would fix [^5]. [^1]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/load/pkg.go#L2768-L2776 [^2]: https://github.com/golang/go/blob/0cd309e12818f988693bf8e4d9f1453331dcf9f2/src/cmd/go/internal/list/list.go#L605-L606 [^3]: golang/go#29666 [^4]: golang/go#49534 (comment) [^5]: #5560 Signed-off-by: Frank Viernau <frank_viernau@epam.com>

sschuberth · 2023-02-23T10:25:09Z

@mawl, can you still reproduce the issue with the latest version of ORT?

fviernau · 2023-02-23T10:32:10Z

@mawl, can you still reproduce the issue with the latest version of ORT?

@sschuberth - This is not supposed to be fixed yet, but it's in the makings. In theory it should be fixed once the next Go version is released and we upgrade to it. See

sschuberth · 2023-04-21T05:58:01Z

Note to myself: The upstream Golang issues have been closed as Gerrit changes with fixes were submitted. These changes did not make it into the 1.20.0 Golang release, but it's a bit hard to figure out whether they have been cherry-picked to any 1.20.x minor release. We should probably just try with a Golang upgrade.

fviernau · 2023-04-21T07:50:27Z

Note to myself: The upstream Golang issues have been closed as Gerrit changes with fixes were submitted. These changes did not make it into the 1.20.0 Golang release, but it's a bit hard to figure out whether they have been cherry-picked to any 1.20.x minor release. We should probably just try with a Golang upgrade.

Here's a first step: #6895

The `GoMod` integration does not execute `go generate/go build`, so the embed directive in `main.go` points to a non-existing file. When `GoMod` executes [1] it failed [2] with the previously used Go version 1.20.5. As of [3], with Go version 1.21.1, that issue is fixed. Add a test to ensure it keeps on working. Note: It is essential to pass `json=Module` instead of not just `-json` [4], because otherwise the issue would still persist on Go 1.21.1. Fixes #5560. [1] `go list -deps -json=Module -buildvcs=false ./...` [2] `pattern file.txt: no matching` [3] 47e4520 [4] 43ad3a7 Signed-off-by: Frank Viernau <frank_viernau@epam.com>

The `GoMod` integration does not execute `go generate/go build`, so the embed directive in `main.go` points to a non-existing file. When `GoMod` executes [1] it failed [2] with the previously used Go version 1.20.5. As of [3], with Go version 1.21.1, that issue is fixed. Add a test to ensure it keeps on working. Note: It is essential to pass `json=Module` instead of not just `-json` [4], because otherwise the issue would still persist on Go 1.21.1. So, [3] and [4] together fix #5560. Fixes #5560. [1] `go list -deps -json=Module -buildvcs=false ./...` [2] `pattern file.txt: no matching` [3] 47e4520 [4] 43ad3a7 Signed-off-by: Frank Viernau <frank_viernau@epam.com>

The `GoMod` integration does not execute `go generate/go build`, so the embed directive in `main.go` points to a non-existing file. When `GoMod` executes [1] it failed [2] with the previously used Go version 1.20.5. As of [3], with Go version 1.21.1, that issue is fixed. Add a test to ensure it keeps on working. Note: It is essential to pass `json=Module` instead of not just `-json` [4], because otherwise the issue would still persist on Go 1.21.1. So, [3] and [4] together fix [5]. Fixes #5560. [1] `go list -deps -json=Module -buildvcs=false ./...` [2] `pattern file.txt: no matching` [3] 47e4520 [4] 43ad3a7 [5] #5560. Signed-off-by: Frank Viernau <frank_viernau@epam.com>

mawl changed the title ~~GoMod: go list -deps error~~ GoMod: go list -deps error: no matching files found Jul 14, 2022

mawl mentioned this issue Jul 14, 2022

GoMod analysis seems to get package versions and graph pruning wrong #5021

Closed

tsteenbe assigned fviernau Jul 14, 2022

fviernau mentioned this issue Feb 14, 2023

GoMod: Upgrade and migrate from go list -f to go list -json #6485

Merged

sschuberth added analyzer About the analyzer tool needs info An issue where further information is required labels Feb 23, 2023

fviernau mentioned this issue Apr 21, 2023

build(Dockerfile): Upgrade Go to version 1.20.3 #6895

Merged

fviernau mentioned this issue Oct 6, 2023

fix(analyzer): Add a test for dangling embed directives / GoMod #7647

Merged

fviernau closed this as completed in #7647 Oct 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GoMod: go list -deps error: no matching files found #5560

GoMod: go list -deps error: no matching files found #5560

mawl commented Jul 14, 2022 •

edited

Loading

fviernau commented Jul 14, 2022 •

edited

Loading

mawl commented Jul 14, 2022

fviernau commented Jul 14, 2022

mawl commented Jul 19, 2022 •

edited

Loading

sschuberth commented Feb 23, 2023

fviernau commented Feb 23, 2023 •

edited

Loading

sschuberth commented Apr 21, 2023

fviernau commented Apr 21, 2023

GoMod: go list -deps error: no matching files found #5560

GoMod: go list -deps error: no matching files found #5560

Comments

mawl commented Jul 14, 2022 • edited Loading

fviernau commented Jul 14, 2022 • edited Loading

mawl commented Jul 14, 2022

fviernau commented Jul 14, 2022

mawl commented Jul 19, 2022 • edited Loading

sschuberth commented Feb 23, 2023

fviernau commented Feb 23, 2023 • edited Loading

sschuberth commented Apr 21, 2023

fviernau commented Apr 21, 2023

mawl commented Jul 14, 2022 •

edited

Loading

fviernau commented Jul 14, 2022 •

edited

Loading

mawl commented Jul 19, 2022 •

edited

Loading

fviernau commented Feb 23, 2023 •

edited

Loading