Linking with clang on aarch64-linux-android results in undefined symbols

[snowp]

We have a fairly large Bazel build which generates a .so for use with Android. Incorporating rules_rust into the mix (even as simple as adding a single rust_library with no addtional dependencies causes the resulting .so to contain two undefined symbols:

                 U _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
                 U _ZN6memchr6memchr8fallback7memchr217h3fdce6c95123ced9E

Looking at one of them, I tracked these down to being referenced and defined by the vendored rustlibs:

Referenced here:

➜  rustlib objdump -x aarch64-linux-android/lib/libobject-e4d8140a90db47ce.rlib|grep _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
0000000000000000         *UND*	0000000000000000 _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
000000000000003c R_AARCH64_CALL26         _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
000000000000001c R_AARCH64_CALL26         _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE

Defined here:

➜  rustlib objdump -d aarch64-linux-android/lib/libmemchr-7409783eefd1e836.rlib|grep _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
Disassembly of section .text._ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE:
0000000000000000 <_ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE>:
       4: 62 01 00 54  	b.hs	0x30 <_ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE+0x30>
       8: 02 09 00 b4  	cbz	x2, 0x128 <_ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE+0x128>

From what I can tell, the linker decides to include the reference (libobject) but does not include the definition (libmemchr).

Looking at the linker args nothing special is done wrt these libraries:

external/androidndk/ndk/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang
-shared
-o
-Wl,-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/external/envoy_mobile/library/common/jni/libenvoy_jni_lib.lo
-Wl,-no-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/external/envoy_mobile/library/common/jni/libndk_jni_support.a
-Wl,-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/src/cc/libjni_interface.lo
-Wl,-no-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/src/cc/libjni_interface_rs-131865085.a
-Wl,-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/src/cc/librust_alloc.lo
-Wl,-no-whole-archive
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/librustc_std_workspace_alloc-e69adea976218243.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/librustc_std_workspace_core-ed1c6adb8e36cabe.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/librustc_std_workspace_std-4294fd5280257638.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libstd-af8cca4ce4f6804f.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libstd_detect-0e975b57636672c8.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libaddr2line-e5d6d51ed5bbf971.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libcfg_if-a5e658c5749822f1.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libgetopts-f5d93f2c3e5a88d2.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libgimli-70667b4716457f81.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libhashbrown-82f9c426c72474a5.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/liblibc-cf442dcec02ecfaf.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libmemchr-7409783eefd1e836.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libminiz_oxide-0e0429dadfd01176.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libobject-e4d8140a90db47ce.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libpanic_unwind-d20b7629bade7b19.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libproc_macro-fb08dac290c4c7f1.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libprofiler_builtins-09579e987bfb7182.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/librustc_demangle-e513a68fe4301435.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libtest-97ee2a167210714e.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libunicode_width-4d4139559e030181.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libunwind-5d21e2a63fe6a0f1.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libadler-f1884c9b41122156.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libcore-669955cdd70efb76.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/libcompiler_builtins-86e41092bca9bdf8.a
bazel-out/android-arm64-v8a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/aarch64-linux-android/lib/liballoc-0a2707a8771b9442.a
<snip, tons more linker args>

I was able to get things working by force linking the stdlib libraries (with the exclusion of the profiler, which resulted in issues resolving the __start___llvm_prf_vnds during dlopen):

diff --git a/rust/toolchain.bzl b/rust/toolchain.bzl
index 687e0df..d4cc7fc 100644
--- a/rust/toolchain.bzl
+++ b/rust/toolchain.bzl
@@ -92,6 +92,7 @@ def _ltl(library, ctx, cc_toolchain, feature_configuration):
         actions = ctx.actions,
         feature_configuration = feature_configuration,
         cc_toolchain = cc_toolchain,
+        alwayslink = ("profiler" not in library.basename),
         static_library = library,
         pic_static_library = library,
     )

So with all this: I'm not sure what the underlying reason behind the linker omitting libmemchr is, nor whether force linking the rustlibs when compiling with a CC toolchain is the right thing to do here. Digging into memchr it seems like perhaps we shouldn't even be using the fallback calls when libc is available, so perhaps this is an issue with the vendored rustlib on aarch64-linux-android.

For reference I have not verified whether this works on other Android platforms, and this is happening rules_rust 0.2.1

[keith]

Here's a repro case for this issue: rustandroidrepro.zip

You must have the android SDK setup for this, but then you can run bazel build main and see the undefined symbols that are an issue:

% nm -u bazel-bin/libmain.so | grep -v LIBC
                 U _ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE
                 U _ZN6memchr6memchr8fallback7memchr217h3fdce6c95123ced9E
                 w bsd_signal
                 w copy_file_range
                 U log
                 U logf

You can workaround this by passing -Wl,--undefined=_ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE (note I think the symbol changes based on some criteria, so this isn't stable).

Thee actual issue here is the order that rules_rust passes the libraries to the linker:

 bazel-out/arm64-v8a-fastbuild/bin/external/rust_darwin_aarch64/lib/rustlib/aarch64-linux-android/lib/libmemchr-7409783eefd1e836.a \
 bazel-out/arm64-v8a-fastbuild/bin/external/rust_darwin_aarch64/lib/rustlib/aarch64-linux-android/lib/libminiz_oxide-0e0429dadfd01176.a \
 bazel-out/arm64-v8a-fastbuild/bin/external/rust_darwin_aarch64/lib/rustlib/aarch64-linux-android/lib/libobject-e4d8140a90db47ce.a \

The issue is the bfd / gold don't go back to discover undefined symbols from archives that were already referenced in the command line, and in this case using -Wl,--trace=_ZN6memchr6memchr8fallback6memchr17h69d58a5ccf33f3aaE you can see that the reference to memchr is from libobject, which is loaded later:

bazel-out/android-armeabi-v7a-fastbuild/bin/external/rust_darwin_x86_64/lib/rustlib/armv7-linux-androideabi/lib/libobject-b92cfab056efaf60.a(object-b92cfab056efaf60.object.44bfcd39-cgu.0.rcgu.o): reference to _ZN6memchr6memchr8fallback6memchr17he975a994fdb39ed4E

You can fix this by passing -fuse-ld=lld for android builds, which is safe and android is generally transitioning to only use lld in the future. We may also consider reordering this for other consumers of rules_rust so they don't also hit this issue.