Gerrit failing with Bazel@HEAD

[meteorcloudy]

https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/434#d0d71269-d29b-41d3-9558-923637bc4c91

ERROR: /var/lib/buildkite-agent/builds/buildkite-worker-ubuntu1804-java9-wr71-1/bazel-downstream-projects/gerrit/java/com/google/gerrit/common/BUILD:10:1: Couldn't build file java/com/google/gerrit/common/libannotations-hjar.jar: Compiling Java headers java/com/google/gerrit/common/libannotations-hjar.jar (3 files) failed (Exit 1)
--
  | An exception has occurred in turbine.
  | java.lang.IllegalArgumentException: Could not find java.lang on bootclasspath
  | at com.google.turbine.binder.Binder.bindPackages(Binder.java:170)
  | at com.google.turbine.binder.Binder.bind(Binder.java:94)
  | at com.google.turbine.main.Main.compile(Main.java:97)
  | at com.google.devtools.build.java.turbine.Turbine.compile(Turbine.java:76)
  | at com.google.devtools.build.java.turbine.Turbine.compile(Turbine.java:69)
  | at com.google.devtools.build.java.turbine.Turbine.main(Turbine.java:46)

/cc @philwo Can you take a look at this?

[aehlig]

Note (see also discussion on #5963):

• only on JDK9 (and Gerrit is the only downstream project we test with JDK9)
• Turbine has been recently updated in commit c1d8776

[cushon]

The only JDK 9 gerrit build I see is from the latest build. Did we just add that configuration to buildkite? It's not clear how recent a regression this is.

[cushon]

The culprit is 7eb9ea1, so I think this also affects 0.17x.

I tested with gerrit at @ 7a06549cdffffab00991ed58f55a97546dbe94ac, and:

bazel build \
  --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
  --java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
  --incompatible_package_name_is_a_function=false //:release

[cushon]

@lberki I mailed you a fix.

[philwo]

Did we just add that configuration to buildkite?

I added this 4 days ago (bazelbuild/continuous-integration@475ccf4). It was requested by @davido and has been green on the Gerrit pipeline (https://buildkite.com/bazel/gerrit) since then, signalling that this would be a regression from the latest release to the next release if it's red now on the downstream project pipeline.

[cushon]

(I pasted the wrong culprit commit above. I edited my comment, but it's 7eb9ea1.)

@philwo does that make sense to you? 7eb9ea1 was submitted four days ago and was cherry-picked into 0.17.

[davido]

It was requested by @davido and has been green on the Gerrit pipeline (https://buildkite.com/bazel/gerrit) since then[...]

Indeed, in the past we have had breakages for building Gerrit with different JDKs combinations, that were already known to work in earlier Bazel releases, without discovering the breakages for quite some time (the last one was: #5961). That why we have added this request (bazelbuild/continuous-integration#173). Having the Bazel-CI for JDK 8|9|10|11|... for Gerrit project would help to avoid regressions in future Bazel releases.

@philwo I've added this issue bazelbuild/continuous-integration#325 to Bazel-CI project to also add CI-Job for Gerrit for Java 10.

[davido]

I confirmed that problem locally. In fact, on Java 9, gerrit build is broken on Bazel 0.17.1rc5

  $ bazel version
Build label: 0.17.1rc5
  $JAVA_HOME/bin/java -fullversion
openjdk full version "9.0.4+12-suse-4.2-x8664"
  $ bazel build \
>   --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
>   --java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
>   --incompatible_package_name_is_a_function=false //:release
INFO: Build options have changed, discarding analysis cache.
INFO: Analysed target //:release (416 packages loaded).
INFO: Found 1 target...
INFO: From Executing genrule @bazel_tools//tools/jdk:platformclasspath9 [for host]:
warning: [options] bootstrap class path not set in conjunction with -source 1.8
1 warning
ERROR: /home/davido/projects/gerrit2/java/com/google/gerrit/common/BUILD:10:1: Compiling Java headers java/com/google/gerrit/common/libannotations-hjar.jar (3 files) failed (Exit 1)
Fatal Error: Unable to find package java.lang in classpath or bootclasspath
Target //:release failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 15.614s, Critical Path: 5.00s
INFO: 23 processes: 1 remote cache hit, 5 linux-sandbox, 17 worker.
FAILED: Build did NOT complete successfully

[cushon]

@davido that's consistent with #6127 (comment), right?

If you want to verify the pending fix: https://bazel-review.googlesource.com/c/bazel/+/74550

[davido]

@cushon Yes, it sounds reasonable. I also confirmed, that on Bazel master (475f40a) there is a different error message (that seems to be identical with the one reported on Bazel CI), so that upcoming 0.18 release is affected as well:

$ bazel build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 --java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 --incompatible_package_name_is_a_function=false //:release
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:98:7: 
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:99:7: Make sure that you are running at least Bazel 0.14.0.
INFO: Analysed target //:release (418 packages loaded).
INFO: Found 1 target...
INFO: From Executing genrule @bazel_tools//tools/jdk:platformclasspath9 [for host]:
warning: [options] bootstrap class path not set in conjunction with -source 1.8
1 warning
ERROR: /home/davido/projects/gerrit2/java/com/google/gerrit/index/BUILD:8:1: Compiling Java headers java/com/google/gerrit/index/libquery_exception-hjar.jar (2 files) failed (Exit 1)
An exception has occurred in turbine.
java.lang.IllegalArgumentException: Could not find java.lang on bootclasspath
	at com.google.turbine.binder.Binder.bindPackages(Binder.java:170)
	at com.google.turbine.binder.Binder.bind(Binder.java:94)
	at com.google.turbine.main.Main.compile(Main.java:97)
	at com.google.devtools.build.java.turbine.Turbine.compile(Turbine.java:76)
	at com.google.devtools.build.java.turbine.Turbine.compile(Turbine.java:69)
	at com.google.devtools.build.java.turbine.Turbine.main(Turbine.java:46)
Target //:release failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 22.342s, Critical Path: 4.75s
INFO: 34 processes: 31 linux-sandbox, 3 worker.
FAILED: Build did NOT complete successfully

I confirmed, that with your CL applied on Bazel master, build is green again on JDK9:

  $ b9 build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 --java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 --incompatible_package_name_is_a_function=false java/com/google/gerrit/index:query_exception
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:98:7: 
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:99:7: Make sure that you are running at least Bazel 0.14.0.
INFO: Analysed target //java/com/google/gerrit/index:query_exception (10 packages loaded).
INFO: Found 1 target...
Target //java/com/google/gerrit/index:query_exception up-to-date:
  bazel-bin/java/com/google/gerrit/index/libquery_exception.jar
INFO: Elapsed time: 7.024s, Critical Path: 4.58s
INFO: 2 processes: 1 linux-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions

[cushon]

@davido thanks for verifying! That is consistent with my understanding of the bug: 7eb9ea1 is present in 0.18 and at head. Once the fix is merged I think the plan is to cherry-pick it into 0.17 and 0.18.

[davido]

Once the fix is merged I think the plan is to cherry-pick it into 0.17 and 0.18.

SGTM.

@cushon Just to make sure I'm not missing something and avoid yet another regression before CI-Job mentioned in bazelbuild/continuous-integration#325 is up and running, I went ahead and tested building Gerrit master (85bf208c81944f2cc91093c6e38be718c4dbeae0) with this pending CL applied: [1] on JDK10 on Bazel@HEAD with your CL applied. It's failing:

  $ java -fullversion
openjdk full version "10.0.2+13-suse-1.1-x8664"
  $ b9 build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java10 --java_toolchain=@bazel_tools//tools/jdk:toolchain_java10 --incompatible_package_name_is_a_function=false :release
WARNING: Running Bazel server needs to be killed, because the startup options are different.
Starting local Bazel server and connecting to it...
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:98:7: 
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:99:7: Make sure that you are running at least Bazel 0.14.0.
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/jdk/BUILD:297:1: every rule of type java_import implicitly depends upon the target '@bazel_tools//tools/jdk:toolchain_java10', but this target could not be found because of: no such target '@bazel_tools//tools/jdk:toolchain_java10': target 'toolchain_java10' not declared in package 'tools/jdk' (did you mean 'toolchain_java8'?) defined by /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/jdk/BUILD
ERROR: /home/davido/projects/gerrit2/java/BUILD:1:1: every rule of type java_binary implicitly depends upon the target '@bazel_tools//tools/jdk:toolchain_java10', but this target could not be found because of: no such target '@bazel_tools//tools/jdk:toolchain_java10': target 'toolchain_java10' not declared in package 'tools/jdk' (did you mean 'toolchain_java8'?) defined by /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/jdk/BUILD
ERROR: Analysis of target '//:release' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.233s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (124 packages loaded)
    currently loading: @com_google_javascript_closure_library//

Shoud I file new issue? Note, that it worked already, as can be seen from this issue: #5967.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/193832

[cushon]

target 'toolchain_java10' not declared

@davido that is the expected result of 7eb9ea1 / #6035. The embedded JDK version was rolled back to JDK 9, and the bundled javac version was rolled back to javac 9, in an effort to stabilize 0.17. Doing so regressed Java 10 support.

[davido]

@cushon Thanks for confirming, I noticed that, yes, but my understanding was, that the efforts are underway to push JDK 10 support back in upcoming 0.18 release again, not? Is it going to be only available in 0.19 or later?

[cushon]

cc @lberki

efforts are underway to push JDK 10 support back in upcoming 0.18 release again, not? Is it going to be only available in 0.19 or later?

I had thought earlier we might be able to roll forward for 0.18, but that's not going to happen. However the fix for #6105 did make it into 0.18, and that change significantly reduces the risk of upgrading to JDK 10 (or JDK 11) in a subsequent release. I'm still hopeful we can restore JDK 10 support in 0.19.

[davido]

@cushon Thanks for confirming and well that's unfortunate. But then we could downgrade the full support for building with Java 10 for gerrit to only supporting building with Vanilla Java Builder on Java 10 on Bazel 0.17 and later.

I updated my CL: [1] and switched to using VanillaJavaBuilder and now building Gerrit on Bazel@HEAD with JDK 10 works again:

  $ bazel build --host_javabase=:absolute_javabase --define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java-10 --host_java_toolchain=//:toolchain_vanilla --java_toolchain=//:toolchain_vanilla :release
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:98:7: 
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:99:7: Make sure that you are running at least Bazel 0.14.0.
INFO: Analysed target //:release (0 packages loaded).
INFO: Found 1 target...
INFO: From Rendering asciidoctor files for html_htmlonly:
unsupported Java version "10", defaulting to 1.7
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.io.FilenoUtil (file:/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/jruby/jar/jruby-complete-9.1.13.0.jar) to method sun.nio.ch.SelChImpl.getFD()
WARNING: Please consider reporting this to the maintainers of org.jruby.util.io.FilenoUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Target //:release up-to-date:
  bazel-bin/release.war
INFO: Elapsed time: 15.732s, Critical Path: 15.43s
INFO: 15 processes: 15 linux-sandbox.
INFO: Build completed successfully, 16 total actions

Update: I applied your suggestion from #6012 and externalized the absolute path to the JDK out of the BUILD file.

• [1] https://gerrit-review.googlesource.com/c/gerrit/+/193832

[laurentlb]

If it affects both 0.17 and 0.18, that's maybe another reason to have a 0.16.2.

[cushon]

The fix was released in 0.17.1.