Builds being killed/memory exhausted on DigitalOcean rhel8-x64 machines

[richardlau]

On both release and test CI builds are unsuccessful on digitalocean-rhel8-x64 machines e.g.

• https://ci.nodejs.org/job/node-test-commit-linux/56965/nodes=rhel8-x64/console on test-digitalocean-rhel8-x64-1:

08:09:25   ccache g++ -o /home/iojs/build/workspace/node-test-commit-linux/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/use-map.o ../deps/v8/src/compiler/turboshaft/use-map.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_X64' '-DV8_HAVE_TARGET_OS' '-DV8_TARGET_OS_LINUX' '-DV8_EMBEDDER_STRING="-node.9"' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT=1' '-DV8_ENABLE_PRIVATE_MAPPING_FORK_OPTIMIZATION' '-DV8_SHORT_BUILTIN_CALLS' '-DOBJECT_PRINT' '-DV8_INTL_SUPPORT' '-DV8_ATOMIC_OBJECT_FIELD_WRITES' '-DV8_ENABLE_LAZY_SOURCE_POSITIONS' '-DV8_USE_SIPHASH' '-DV8_SHARED_RO_HEAP' '-DNDEBUG' '-DV8_WIN64_UNWINDING_INFO' '-DV8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH' '-DV8_USE_ZLIB' '-DV8_ENABLE_SPARKPLUG' '-DV8_ENABLE_MAGLEV' '-DV8_ENABLE_TURBOFAN' '-DV8_ENABLE_WEBASSEMBLY' '-DV8_ENABLE_JAVASCRIPT_PROMISE_HOOKS' '-DV8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA' '-DV8_ALLOCATION_FOLDING' '-DV8_ALLOCATION_SITE_TRACKING' '-DV8_ADVANCED_BIGINT_ALGORITHMS' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../deps/v8/include -I/home/iojs/build/workspace/node-test-commit-linux/out/Release/obj/gen/generate-bytecode-output-root -I/home/iojs/build/workspace/node-test-commit-linux/out/Release/obj/gen -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -Wno-return-type -flax-vector-conversions -Wno-invalid-offsetof -fno-strict-aliasing -m64 -m64 -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -std=gnu++17 -MMD -MF /home/iojs/build/workspace/node-test-commit-linux/out/Release/.deps//home/iojs/build/workspace/node-test-commit-linux/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/use-map.o.d.raw   -c
08:19:47 g++: fatal error: Killed signal terminated program cc1plus
08:19:47 compilation terminated.
08:19:47 make[2]: *** [tools/v8_gypfiles/v8_turboshaft.target.mk:222: /home/iojs/build/workspace/node-test-commit-linux/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/csa-optimize-phase.o] Error 1
08:19:47 make[2]: *** Waiting for unfinished jobs....

• https://ci-release.nodejs.org/job/iojs+release/10086/nodes=rhel8-x64-release/console on release-digitalocean-rhel8-x64-1:

07:10:22   ccache g++ -o /home/iojs/build/ws/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/graph-builder.o ../deps/v8/src/compiler/turboshaft/graph-builder.cc '-D_GLIBCXX_USE_CXX11_ABI=1' '-DNODE_OPENSSL_CONF_NAME=nodejs_conf' '-DNODE_OPENSSL_HAS_QUIC' '-DICU_NO_USER_DATA_OVERRIDE' '-DV8_GYP_BUILD' '-DV8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64' '-D__STDC_FORMAT_MACROS' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-DV8_TARGET_ARCH_X64' '-DV8_HAVE_TARGET_OS' '-DV8_TARGET_OS_LINUX' '-DV8_EMBEDDER_STRING="-node.9"' '-DENABLE_DISASSEMBLER' '-DV8_PROMISE_INTERNAL_FIELD_COUNT=1' '-DV8_ENABLE_PRIVATE_MAPPING_FORK_OPTIMIZATION' '-DV8_SHORT_BUILTIN_CALLS' '-DOBJECT_PRINT' '-DV8_INTL_SUPPORT' '-DV8_ATOMIC_OBJECT_FIELD_WRITES' '-DV8_ENABLE_LAZY_SOURCE_POSITIONS' '-DV8_USE_SIPHASH' '-DV8_SHARED_RO_HEAP' '-DNDEBUG' '-DV8_WIN64_UNWINDING_INFO' '-DV8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH' '-DV8_USE_ZLIB' '-DV8_ENABLE_SPARKPLUG' '-DV8_ENABLE_MAGLEV' '-DV8_ENABLE_TURBOFAN' '-DV8_ENABLE_WEBASSEMBLY' '-DV8_ENABLE_JAVASCRIPT_PROMISE_HOOKS' '-DV8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA' '-DV8_ALLOCATION_FOLDING' '-DV8_ALLOCATION_SITE_TRACKING' '-DV8_ADVANCED_BIGINT_ALGORITHMS' '-DUCONFIG_NO_SERVICE=1' '-DU_ENABLE_DYLOAD=0' '-DU_STATIC_IMPLEMENTATION=1' '-DU_HAVE_STD_STRING=1' '-DUCONFIG_NO_BREAK_ITERATION=0' -I../deps/v8 -I../deps/v8/include -I/home/iojs/build/ws/out/Release/obj/gen/generate-bytecode-output-root -I/home/iojs/build/ws/out/Release/obj/gen -I../deps/icu-small/source/i18n -I../deps/icu-small/source/common -I../deps/v8/third_party/abseil-cpp  -pthread -Wno-unused-parameter -Wno-return-type -flax-vector-conversions -Wno-invalid-offsetof -fno-strict-aliasing -m64 -m64 -O3 -fno-omit-frame-pointer -fdata-sections -ffunction-sections -O3 -fno-rtti -fno-exceptions -std=gnu++17 -MMD -MF /home/iojs/build/ws/out/Release/.deps//home/iojs/build/ws/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/graph-builder.o.d.raw   -c
08:40:19 virtual memory exhausted: Cannot allocate memory
08:40:19 make[2]: *** [tools/v8_gypfiles/v8_turboshaft.target.mk:222: /home/iojs/build/ws/out/Release/obj.target/v8_turboshaft/deps/v8/src/compiler/turboshaft/csa-optimize-phase.o] Error 1
08:40:19 make[2]: *** Waiting for unfinished jobs....

Looking at https://ci.nodejs.org/job/node-test-commit-linux/nodes=rhel8-x64/buildTimeTrend it looks like this is not occurring on the IBM Cloud hosted rhel8-x64 machines.

[richardlau]

When I migrated release-ibm-rhel8-x64-2 from SJC01 I remember noticing that IBM Cloud gave new VMs some swap storage, which might be the difference assuming the DigitalOcean droplets have the same RAM allocated as the IBM Cloud VMs.

[richardlau]

Looks like all of the VMs have 4GB of RAM. The IBM Cloud VMs have 2GB of swap which the DigitalOcean droplets do not.

$ ansible -m command -a "free -h" *-rhel8-x64-*
release-digitalocean-rhel8-x64-1 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       337Mi       2.6Gi       184Mi       602Mi       2.9Gi
Swap:            0B          0B          0B
release-ibm-rhel8-x64-1 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       1.8Gi       1.1Gi        81Mi       926Mi       1.7Gi
Swap:         2.0Gi       240Mi       1.8Gi
release-ibm-rhel8-x64-2 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       1.3Gi       1.5Gi        94Mi       946Mi       2.2Gi
Swap:         2.0Gi       275Mi       1.7Gi
test-digitalocean-rhel8-x64-1 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       440Mi       2.3Gi       192Mi       915Mi       2.8Gi
Swap:            0B          0B          0B
test-ibm-rhel8-x64-2 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       348Mi       1.3Gi        86Mi       2.2Gi       3.1Gi
Swap:         2.0Gi       200Mi       1.8Gi
test-ibm-rhel8-x64-3 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       347Mi       2.7Gi        67Mi       805Mi       3.2Gi
Swap:         2.0Gi       217Mi       1.8Gi
test-ibm-rhel8-x64-1 | CHANGED | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       667Mi       911Mi        57Mi       2.3Gi       2.8Gi
Swap:         2.0Gi       285Mi       1.7Gi
$