Zero byte static library file prevents ar from archiving

[gjasny]

Description of the problem / feature request:

Recently we added Android NDK ARMv7, ARMv8, and x86_64 to our CI. From time to time we see the following errors for the arm flavours (maybe x86_64 was just lucky):

ERROR: /home/ec2-user/workspace/695-no-notify-4-lib-changes-GLLX2W3TUR6VH4VERTJCS3OJYCQU6LY7N4HBAUOVNLUVMRD374HQ/modules/transportengine/Backends/avtp3/BUILD.bazel:34:1: Linking of rule '//modules/transportengine/Backends/avtp3:rtctransportengine_avtp3' failed (Exit 1): aarch64-linux-android-ar failed: error executing command 

  (cd /cache/bzl/out/f6ea00053de6211aeeee07d0232330a2/execroot/rtcplatform && \

  exec env - \

    PATH=/bin:/usr/bin \

    PWD=/proc/self/cwd \

  external/androidndk/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/aarch64-linux-android-ar rcsD bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/librtctransportengine_avtp3.a bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportAudioChannelRx.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportAudioChannelTx.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportConnection.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportEngineBackend.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportEngineBackendFactory.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportMediaChannel.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportVideoRxChannel.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportVideoTxChannel.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpTransportVideoTxChannelSetupRequest.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/AvtpUtils.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/SetupHandlerDataHelper.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IChannelSetupCallbackProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IConnectionCallbackProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IDataChannelCallbackProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IGenericChannelCallbackProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IAvtpIncomingConnectionRequestProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IAvtpTransportAudioChannelRxInternalProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IAvtpTransportAudioChannelTxInternalProxy.o bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/_objs/rtctransportengine_avtp3/IAvtpTransportConnectionInternalProxy.o).

external/androidndk/ndk/toolchains/aarch64-linux-android-4.9/prebuilt/linux-x86_64/bin/aarch64-linux-android-ar: bazel-out/arm64-v8a-dbg/bin/modules/transportengine/Backends/avtp3/librtctransportengine_avtp3.a: File format not recognized

Inspection of the librtctransportengine_avtp3.a file revealed that it is zero bytes and thus lacking a proper header which makes ar bail out with File format not recognized.

Now I wonder how that empty file could appear in the first place?

We're using a remote rest cache that sometimes is overloaded and might refuse or fail to deliver cache objects. I could imagine that upon a cache failure an empty file is left behind. For other cache objects like a C++ compiler invocation that might go unnoticed because the compiler overwrites the target file whereas ar appends output.

Do you have some hints how to debug that issue further? Is bazel actually calling ar to update static libraries or is it always performing a complete re-creation where we could delete the target file before?

What operating system are you running Bazel on?

Ubuntu 16.04 docker container on a AWS Linux 2 host

What's the output of bazel info release?

release 0.16.1

[gjasny]

It is not limited to Android and also happens for Ubuntu builds:

ERROR: /home/ec2-user/workspace/-and-provide-puppeteer-deps-CDC24O45VWEM7U2UMTIEUPK2EGOOCBVNDY3NM7LFMEHRSZCEB67Q/modules/ace/AudioComponents/Test/Node/BUILD.bazel:12:1: Linking of rule '//modules/ace/AudioComponents/Test/Node:Node-ce-protos' failed (Exit 1): ar failed: error executing command 

  (cd /cache/bzl/out/3318f3a592e7ba40b0d45fa930a2e652/execroot/rtcplatform && \

  exec env - \

    PATH=/bin:/usr/bin \

    PWD=/proc/self/cwd \

  /usr/bin/ar @bazel-out/k8-dbg/bin/modules/ace/AudioComponents/Test/Node/libNode-ce-protos.a-2.params).

/usr/bin/ar: bazel-out/k8-dbg/bin/modules/ace/AudioComponents/Test/Node/libNode-ce-protos.a: File format not recognized

[gjasny]

Note: prior to calling bazel build we call bazel clean so the build should start clean.

[jin]

@gjasny does this problem occur when using a local cache?

[gjasny]

We were using a remote cache. But I have not seen the problem since a month. We could close it and I re-open once the problem re-appears.

[jin]

@gjasny sounds good! Will close this for now.