[BEAM-13647] Use role for Go worker binary.

[lostluck]

Move away from older version of Go Worker Binary, in favour of a dedicated RoleUrn.

1. Adds "beam:artifact:role:go_worker_binary:v1" urn to proto.
2. On pipeline execution, updates the default Go env in the pipeline proto with the worker binary, as appropriate:

• Continue to use file references for universal runners
• Start using URL types pointing to GCS staging location for Dataflow.

3. Retain legacy logic in case of old SDKs using newer SDK container for some reason.
4. Newer SDKs can still use older SDK containers as there's no simple mechanism at present to add additional dependencies to the Go environment. Therefore there's often only 1 dependency and it will always be the worker. However, from 2.37.0 on, the role will be usable and expected.

Validated "old container" behavior manually against Spark and Dataflow, with the pipelines successfully executing with each.

Minor breaking change to the dataflow lib, unexporting the StageFile function, and also have it return the SHA256 hash for the binary. dataflowlib was never intended to be used outside of the dataflow runner code anyway.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---					---
XLang						---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #16729 (17e7890) into master (730b941) will increase coverage by 28.17%.
The diff coverage is 46.15%.

@@             Coverage Diff             @@
##           master   #16729       +/-   ##
===========================================
+ Coverage   46.49%   74.67%   +28.17%     
===========================================
  Files         202      654      +452     
  Lines       19940    82133    +62193     
===========================================
+ Hits         9271    61331    +52060     
- Misses       9679    19812    +10133     
  Partials      990      990               

Impacted Files	Coverage Δ
sdks/go/pkg/beam/artifact/materialize.go	48.94% <0.00%> (ø)
...o/pkg/beam/runners/dataflow/dataflowlib/execute.go	0.00% <0.00%> (ø)
.../go/pkg/beam/runners/dataflow/dataflowlib/stage.go	0.00% <0.00%> (ø)
sdks/go/pkg/beam/core/runtime/graphx/translate.go	33.49% <100.00%> (+5.12%)	⬆️
sdks/python/apache_beam/io/mongodbio.py	92.85% <0.00%> (ø)
sdks/python/apache_beam/internal/__init__.py	100.00% <0.00%> (ø)
...on/apache_beam/portability/api/metrics_pb2_grpc.py	0.00% <0.00%> (ø)
sdks/python/apache_beam/utils/annotations.py	100.00% <0.00%> (ø)
...ache_beam/io/gcp/datastore/v1new/query_splitter.py	93.75% <0.00%> (ø)
sdks/python/apache_beam/internal/dill_pickler.py	86.33% <0.00%> (ø)
... and 448 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 730b941...17e7890. Read the comment docs.

[lostluck]

R: @ihji

[lostluck]

Run Go Flink ValidatesRunner

[lostluck]

Run Go Spark ValidatesRunner

[lostluck]

Run Go Samza ValidatesRunner

[ihji]

Thanks, LGTM. Just one minor comment.

[ihji]

Please fix codecov failure before merging it.

[lostluck]

Please fix codecov failure before merging it.

Note: All this code is covered by the integration tests, otherwise it wouldn't work. Adding test coverage to boot.go would require a complete refactor out of the single main method. I'll see what I can do about the other code though, in particular the binary substitution.

[lostluck]

Added tests to the core changes in the graphx package. The other packages changes require various bits of file IO, without which, the SDK wouldn't function at all. They would need to be refactored a bit, or have specialized file-io based tests to validate them, which is a bit much for this change.

[lostluck]

Run Java PreCommit