android_library builds java_library deps with as Java 1.9 since Bazel 0.16

[steeve]

Description of the problem / feature request:

When building an android_library, Bazel builds java_library deps as Java 1.9 since it went to the JDK9.

This results in the following proguard issues when building downstream, for instance with Protobuf Javalite:

Warning: com.google.protobuf.ByteBufferWriter: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$ArrayEncoder: can't find referenced method 'java.nio.ByteBuffer clear()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$ByteOutputEncoder: can't find referenced method 'java.nio.ByteBuffer clear()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$NioEncoder: can't find referenced method 'java.nio.ByteBuffer clear()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$NioEncoder: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$NioHeapEncoder: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.CodedOutputStream$OutputStreamEncoder: can't find referenced method 'java.nio.ByteBuffer clear()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.Internal: can't find referenced method 'java.nio.ByteBuffer clear()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.NioByteString: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.NioByteString: can't find referenced method 'java.nio.ByteBuffer limit(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.NioByteString$1: can't find referenced method 'java.nio.ByteBuffer mark()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.NioByteString$1: can't find referenced method 'java.nio.ByteBuffer reset()' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.Utf8$Processor: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer
Warning: com.google.protobuf.Utf8$UnsafeProcessor: can't find referenced method 'java.nio.ByteBuffer position(int)' in library class java.nio.ByteBuffer

The issue is fixed by manually reverting to JDK8 with the following options:

build:jdk8 --javabase=@local_jdk//:jdk
build:jdk8 --host_javabase=@local_jdk//:jdk
build:jdk8 --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
build:jdk8 --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8

Android does not support java.nio nor JDK9 yet.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Build an Android app with proguard and Protobuf Javalite.

What operating system are you running Bazel on?

OSX

What's the output of bazel info release?

release 0.16.1

Have you found anything relevant by searching the web?

I found an issue that said to manually add --javacopts="-source 8 -target 8". Not sure this is the recommended way until Bazel does it itself ?

Thanks!

[steeve]

Probably because this doesn't check for Java 1.9 https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/android/BazelAndroidSemantics.java#L46

[steeve]

This also results in the following Bazel warnings when building:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.devtools.build.android.desugar.Desugar (file:/private/var/tmp/_bazel_steeve/install/2fc3f6f2633d78815ad4e94b45964b0e/_embedded_binaries/embedded_tools/src/tools/android/java/com/google/devtools/build/android/all_android_tools_deploy.jar) to field java.lang.invoke.InnerClassLambdaMetafactory.dumper
WARNING: Please consider reporting this to the maintainers of com.google.devtools.build.android.desugar.Desugar
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

[cushon]

Does this reproduce with 0.17 or Bazel at head? This might be the same issue as #5888.

[jin]

I can reproduce this problem with building an Android app and RBE, with the following bazelrc config:

https://github.com/bazelbuild/bazel-toolchains/blob/a47750aac6225fdf3f28faea8f378b61eb965786/bazelrc/bazel-0.16.1.bazelrc#L42:L45

[cushon]

@jin which Bazel version you reproduce with? Are you seeing with issue with the 0.17rc and at head, or just at 0.16.1?

[jin]

Just 0.17.1rc1 and HEAD. 0.16.1 is fine.

[cushon]

I think this is still #5888, and the issue is that when Bazel is built at HEAD or 0.17rc using 0.16.1, 0.16.1 produces a Bazel binary that contains the linkage issue.

If I build Bazel at head and then re-build using the HEAD binary (i.e. bazel build //src::bazel && bazel-bin/src/bazel build //src:bazel) then I get a binary that doesn't exhibit this bug.

[jin]

@buchgr @philwo it looks like the release pipeline needs to be updated to build the release Bazel with itself. Can this be done?

Marking this as a release blocker.

[philwo]

@jin I have modified the release pipeline accordingly now.

However, it's worrying that this regression wasn't caught by our integration tests on CI (if I understand correctly). In theory, the commit that caused Bazel 0.16.1 to produce broken binaries should not have been able to be committed, because presubmit tests should have caught it, right?

How can we prevent this in the future?

[jin]

This was only caught when using a custom JDK8 toolchain for RBE to build an Android app. The incompatible code path was in the Android build logic. Perhaps we can build //examples/... in the RBE pipeline?

…

On Aug 27, 2018, at 7:31 AM, Philipp Wollermann ***@***.***> wrote: @jin I have modified the release pipeline accordingly now. However, it's worrying that this regression wasn't caught by our integration tests on CI (if I understand correctly). In theory, the commit that caused Bazel 0.16.1 to produce broken binaries should not have been able to be committed, because presubmit tests should have caught it, right? How can we prevent this in the future? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[steeve]

Catching up with the issue, it's great you guys found the issue!

[philwo]

Perhaps we can build //examples/... in the RBE pipeline?

I don't know - can we? :D From a config perspective, we just have to add it here:

https://github.com/bazelbuild/bazel/blob/master/.bazelci/presubmit.yml#L120
https://github.com/bazelbuild/bazel/blob/master/.bazelci/postsubmit.yml#L93

Can you try if it works and send me a CL with the changes?

[jin]

Opened #6007

[laurentlb]

Please let me know if there's a cherrypick to do in Bazel release.

[jin]

I don’t think there is one, we just needed a release pipeline update (thanks @philwo!). We can close this issue after verifying that the new RC fixes it.

…

On Aug 28, 2018, at 8:56 AM, Laurent Le Brun ***@***.***> wrote: Please let me know if there's a cherrypick to do in Bazel release. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jin]

I can verify that building the latest bazel with the latest rc (0.17.1rc1) fixes this issue. @steeve, please reopen if you are still running into this issue.

[steeve]

Hey guys, sorry I report I'm still running into this issue with 0.17.1 (sorry I couldn't test earlier).

However this time the bazelrc workaround doesn't work:

ERROR: /private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/external/com_android_support_support_v4/BUILD.bazel:2:1: Desugaring external/com_android_support_support_v4/_aar/library/classes_and_libs_merged.jar for Android failed (Exit 1): desugar_java8 failed: error executing command
  (cd /private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/execroot/__main__ && \
  exec env - \
    PATH=/bin:/usr/bin \
  bazel-out/host/bin/external/bazel_tools/tools/android/desugar_java8 @bazel-out/android-arm64-v8a-fastbuild/bin/external/com_android_support_support_v4/_dx/library/classes_and_libs_merged.jar_desugared.jar-0.params)
Unrecognized option: --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

[cushon]

@steeve can you provide steps for reproducing the problem you're seeing?

[steeve]

@cushon I put together a repro repo for both scenarios: https://github.com/steeve/bazel_issue_5978

[cushon]

@steeve thanks! I started a new issue for this: #6159.

[steeve]

Thanks!
Although I'm not sure we are supposed to force JDK8 for building Android apps are we ?

Without forcing JDK8 it "works", but:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/private/var/tmp/_bazel_steeve/install/79d9f22886570ca416e6ffca28ab7725/_embedded_binaries/embedded_tools/src/tools/android/java/com/google/devtools/build/android/all_android_tools_deploy.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Ultimately this won't work because Android doesn't support JDK >8.

I'm think we might want to keep this issue open, wether it's in the form of documentation (to tell people to force using jdk8) or fixing it altogether.