You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The recent bump to Ubuntu 20.04 in #14432 caused a version mismatch in glibc (among probably many other things) between the Azure runners and the self-hosted runners. The remote cache is not able to differentiate builds between these two platforms, leading to cache hits where there should not be. This is leading to our self-hosted runners (using Ubuntu 18.04) pulling cached artifacts that were built on Azure runners (using Ubuntu 20.04).
The immediate solution is going to be sharding the cache by including the OS version in the build actions, similarly to what's described here. In the short term, we need to prioritize upgrading the GCP nodes and the FPGA runner to Ubuntu 20.04. In the long term, making the entire build process hermetic will solve this problem entirely.
The text was updated successfully, but these errors were encountered:
Testing //.../rom/e2e:e2e_bootstrap_entry_fpga_cw310; 0s local
==================== Test output for //sw/device/silicon_creator/rom/e2e:e2e_bootstrap_entry_fpga_cw310:
Invoking test: sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry --cw310-uarts=/dev/ttyACM_CW310_1,/dev/ttyACM_CW310_0 --rcfile= --logging=info --interface=cw310 --rom-kind=rom --bitstream=external/bitstreams/cache/7cbb64c42451e388246c37f46252db1fbd8cd550/lowrisc_systems_chip_earlgrey_cw310_0.1.bit.splice --bootstrap=sw/device/silicon_creator/rom/e2e/e2e_bootstrap_entry_prog_fpga_cw310.bin
sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry)
sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry)
sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by sw/host/tests/rom/e2e_bootstrap_entry/e2e_bootstrap_entry)
================================================================================
The recent bump to Ubuntu 20.04 in #14432 caused a version mismatch in glibc (among probably many other things) between the Azure runners and the self-hosted runners. The remote cache is not able to differentiate builds between these two platforms, leading to cache hits where there should not be. This is leading to our self-hosted runners (using Ubuntu 18.04) pulling cached artifacts that were built on Azure runners (using Ubuntu 20.04).
This is a known issue mentioned in the Remote Caching documentation and is tracked in bazelbuild/bazel#4558.
The immediate solution is going to be sharding the cache by including the OS version in the build actions, similarly to what's described here. In the short term, we need to prioritize upgrading the GCP nodes and the FPGA runner to Ubuntu 20.04. In the long term, making the entire build process hermetic will solve this problem entirely.
The text was updated successfully, but these errors were encountered: