Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when taking a heap snapshot #38985

Closed
jaysoffian opened this issue Jun 10, 2021 · 10 comments
Closed

Crash when taking a heap snapshot #38985

jaysoffian opened this issue Jun 10, 2021 · 10 comments
Labels
inspector Issues and PRs related to the V8 inspector protocol v8 module Issues and PRs related to the "v8" subsystem.

Comments

@jaysoffian
Copy link

jaysoffian commented Jun 10, 2021 •
edited
Loading

  • Version: v14.17.0
  • Platform: Darwin C02XF1E9JHD3 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64
  • Subsystem: heap snapshot

What steps will reproduce the bug?

I have a node process that crashes fairly consistently when taking a heap snapshot via Chrome dev tools. Here's a stack trace captured by lldb:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x80ec05)
    frame #0: 0x000000010068c1eb node`v8::internal::V8HeapExplorer::ExtractContextReferences(v8::internal::HeapEntry*, v8::internal::Context) + 379
node`v8::internal::V8HeapExplorer::ExtractContextReferences:
->  0x10068c1eb <+379>: cmpq   -0x9480(%rdx), %rcx
    0x10068c1f2 <+386>: je     0x10068c210               ; <+416>
    0x10068c1f4 <+388>: movq   0x1f(%rax), %rcx
    0x10068c1f8 <+392>: leaq   0xd6c4b1(%rip), %rdx      ; "extension"
Target 0: (node) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x80ec05)
  * frame #0: 0x000000010068c1eb node`v8::internal::V8HeapExplorer::ExtractContextReferences(v8::internal::HeapEntry*, v8::internal::Context) + 379
    frame #1: 0x000000010068e0cc node`v8::internal::V8HeapExplorer::IterateAndExtractReferences(v8::internal::HeapSnapshotGenerator*) + 956
    frame #2: 0x0000000100690653 node`v8::internal::HeapSnapshotGenerator::GenerateSnapshot() + 211
    frame #3: 0x0000000100684eab node`v8::internal::HeapProfiler::TakeSnapshot(v8::ActivityControl*, v8::HeapProfiler::ObjectNameResolver*, bool) + 107
    frame #4: 0x00000001008eca0d node`v8_inspector::V8HeapProfilerAgentImpl::takeHeapSnapshot(v8_crdtp::glue::detail::ValueMaybe<bool>, v8_crdtp::glue::detail::ValueMaybe<bool>) + 301
    frame #5: 0x00000001009ca0a6 node`v8_inspector::protocol::HeapProfiler::DomainDispatcherImpl::takeHeapSnapshot(v8_crdtp::Dispatchable const&, v8_inspector::protocol::DictionaryValue*, v8_crdtp::ErrorSupport*) + 374
    frame #6: 0x00000001009ca8f9 node`std::__1::__function::__func<v8_inspector::protocol::HeapProfiler::DomainDispatcherImpl::Dispatch(v8_crdtp::span<unsigned char>)::$_0, std::__1::allocator<v8_inspector::protocol::HeapProfiler::DomainDispatcherImpl::Dispatch(v8_crdtp::span<unsigned char>)::$_0>, void (v8_crdtp::Dispatchable const&)>::operator()(v8_crdtp::Dispatchable const&) + 137
    frame #7: 0x0000000100922528 node`v8_crdtp::UberDispatcher::DispatchResult::Run() + 24
    frame #8: 0x00000001008f6792 node`v8_inspector::V8InspectorSessionImpl::dispatchProtocolMessage(v8_inspector::StringView) + 562
    frame #9: 0x00000001001852ee node`node::inspector::NodeInspectorClient::dispatchMessageFromFrontend(int, v8_inspector::StringView const&) + 302
    frame #10: 0x0000000100184f3a node`node::inspector::(anonymous namespace)::SameThreadInspectorSession::Dispatch(v8_inspector::StringView const&) + 58
    frame #11: 0x000000010019795b node`node::inspector::(anonymous namespace)::MainThreadSessionState::Dispatch(std::__1::unique_ptr<v8_inspector::StringBuffer, std::__1::default_delete<v8_inspector::StringBuffer> >) + 43
    frame #12: 0x000000010019799e node`void node::inspector::(anonymous namespace)::AnotherThreadObjectReference<node::inspector::(anonymous namespace)::MainThreadSessionState>::Apply<std::__1::unique_ptr<v8_inspector::StringBuffer, std::__1::default_delete<v8_inspector::StringBuffer> > >(node::inspector::(anonymous namespace)::MainThreadSessionState*, void (node::inspector::(anonymous namespace)::MainThreadSessionState::*)(std::__1::unique_ptr<v8_inspector::StringBuffer, std::__1::default_delete<v8_inspector::StringBuffer> >), std::__1::unique_ptr<v8_inspector::StringBuffer, std::__1::default_delete<v8_inspector::StringBuffer> >&) + 46
    frame #13: 0x000000010019656d node`node::inspector::MainThreadInterface::DispatchMessages() + 365
    frame #14: 0x0000000100197fde node`node::CallbackQueue<void, node::Environment*>::CallbackImpl<node::inspector::MainThreadInterface::Post(std::__1::unique_ptr<node::inspector::Request, std::__1::default_delete<node::inspector::Request> >)::$_0>::Call(node::Environment*) + 46
    frame #15: 0x00000001000618d1 node`node::Environment::RunAndClearInterrupts() + 65
    frame #16: 0x00000001000654ef node`node::Environment::RequestInterruptFromV8()::$_8::__invoke(v8::Isolate*, void*) + 31
    frame #17: 0x0000000100343c55 node`v8::internal::Isolate::InvokeApiInterruptCallbacks() + 293
    frame #18: 0x000000010035b2b1 node`v8::internal::StackGuard::HandleInterrupts() + 1633
    frame #19: 0x00000001006fb7b7 node`v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) + 519
    frame #20: 0x0000000100a893d9 node`Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit + 57
    frame #21: 0x00002e8cf0709aaf
    frame #22: 0x0000000100a1ffba node`Builtins_JSEntryTrampoline + 90
    frame #23: 0x0000000100a1fd98 node`Builtins_JSEntry + 120
    frame #24: 0x0000000100334e98 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 3048
    frame #25: 0x00000001003342a2 node`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 210
    frame #26: 0x0000000100212b23 node`v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 435
    frame #27: 0x0000000100002240 node`node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) + 688
    frame #28: 0x00000001000163f8 node`node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) + 200
    frame #29: 0x00000001001caba4 node`node::TLSWrap::SSLInfoCallback(ssl_st const*, int, int) + 468
    frame #30: 0x0000000100b78150 node`tls_finish_handshake + 496
    frame #31: 0x0000000100b6cf7a node`state_machine + 1274
    frame #32: 0x0000000100b3ed78 node`ssl3_write_bytes + 328
    frame #33: 0x0000000100b57be8 node`SSL_write + 24
    frame #34: 0x00000001001cc3c6 node`node::TLSWrap::ClearIn() + 310
    frame #35: 0x00000001001cddd8 node`node::TLSWrap::OnStreamRead(long, uv_buf_t const&) + 136
    frame #36: 0x000000010016ca2b node`node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) + 699
    frame #37: 0x0000000100a0f46e node`uv__stream_io + 1870
    frame #38: 0x0000000100a180ac node`uv__io_poll + 2060
    frame #39: 0x0000000100a049a1 node`uv_run + 401
    frame #40: 0x00000001000f2075 node`node::NodeMainInstance::Run() + 309
    frame #41: 0x00000001000851b6 node`node::Start(int, char**) + 294
    frame #42: 0x00007fff203e3f5d libdyld.dylib`start + 1

(Process started using lldb -- node --inspect index.js.)

How often does it reproduce? Is there a required condition?

This process runs tasks periodically via npm package cron. The crash seems to be more likely to occur while the node process is actively running a task.

What is the expected behavior?

Not to crash.

What do you see instead?

Crash.

Additional information

Roughly, this code periodically uses the got package to query data from a couple different APIs, then insert the data into MySQL using the mysql2 package. As mentioned above, the crash is more likely to occur while the process is actively querying data and inserting it into MySQL. Running it under lldb seems to further increase the likelihood of the crash occuring.

The Chrome version is 91.0.4472.77 with built-in dev tools.

Node installed via nvm 0.38.0.
@jaysoffian jaysoffian changed the title Crash when in inspector take a heap snapshot Crash when taking a heap snapshot Jun 10, 2021
@Ayase-252 Ayase-252 added the inspector Issues and PRs related to the V8 inspector protocol label Jun 10, 2021
@joyeecheung
Copy link
Member

This seems to be similar to #37878 and #38961, all crashed at ExtractContextReferences, and with only descriptive information of what code was being run when the crash happened so it is difficult to reproduce

@joyeecheung joyeecheung added the v8 module Issues and PRs related to the "v8" subsystem. label Jun 10, 2021
@joyeecheung
Copy link
Member

Is it possible to provide some code that can be run to reproduce this?

@jaysoffian
Copy link
Author

This is commercial code so unfortunately I cannot provide it. I don't currently have the time to work on a minimal reproducer. It pretty reliably crashes when taking a snapshot while the process is active. Is there any other data I can provide beyond the code and backtrace?

@bl-ue
Copy link
Contributor

bl-ue commented Jun 10, 2021

Unfortunately, I'm afraid that we can't help you debug much if we can't reproduce it ourselves 😞

@jaysoffian
Copy link
Author

jaysoffian commented Jun 10, 2021 •
edited
Loading

Also segfaults when running the code one-shot when heap profiling "Allocation instrumentation on timeline" and "Record stack traces of allocations (extra performance overhead)" checked. This crashes 100% of the time several minutes into the run with:

Process 51848 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffbffffffff)
    frame #0: 0x0000000100680844 node`v8::internal::AllocationTracker::AllocationEvent(unsigned long, int) + 164
node`v8::internal::AllocationTracker::AllocationEvent:
->  0x100680844 <+164>: movq   -0x1(%r15), %rsi
    0x100680848 <+168>: decq   %r15
    0x10068084b <+171>: leaq   -0x48(%rbp), %rdi
    0x10068084f <+175>: callq  0x1005dc350               ; v8::internal::HeapObject::SizeFromMap(v8::internal::Map) const
Target 0: (node) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffbffffffff)
  * frame #0: 0x0000000100680844 node`v8::internal::AllocationTracker::AllocationEvent(unsigned long, int) + 164
    frame #1: 0x000000010037c82e node`v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 574
    frame #2: 0x00000001003ae7c1 node`v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 33
    frame #3: 0x00000001003ae861 node`v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 33
    frame #4: 0x0000000100375778 node`v8::internal::FactoryBase<v8::internal::Factory>::NewByteArray(int, v8::internal::AllocationType) + 56
    frame #5: 0x0000000100318b02 node`v8::internal::TranslatedState::EnsureJSObjectAllocated(v8::internal::TranslatedValue*, v8::internal::Handle<v8::internal::Map>) + 114
    frame #6: 0x00000001003181cb node`v8::internal::TranslatedState::EnsureCapturedObjectAllocatedAt(int, std::__1::stack<int, std::__1::deque<int, std::__1::allocator<int> > >*) + 811
    frame #7: 0x0000000100313b76 node`v8::internal::TranslatedState::EnsureObjectAllocatedAt(v8::internal::TranslatedValue*) + 374
    frame #8: 0x0000000100310b7e node`v8::internal::TranslatedValue::GetValue() + 238
    frame #9: 0x0000000100310919 node`v8::internal::Deoptimizer::MaterializeHeapObjects() + 265
    frame #10: 0x00000001006e44fe node`v8::internal::Runtime_NotifyDeoptimized(int, unsigned long*, v8::internal::Isolate*) + 430
    frame #11: 0x0000000100a893d9 node`Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit + 57
    frame #12: 0x0000000100a22ef4 node`Builtins_NotifyDeoptimized + 20
    frame #13: 0x0000000100a228a0 node`Builtins_InterpreterEnterBytecodeAdvance + 224
    frame #14: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #15: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #16: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #17: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #18: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #19: 0x0000000100aa4fd2 node`Builtins_ArrayMap + 1010
    frame #20: 0x0000000100a222a2 node`Builtins_InterpreterEntryTrampoline + 194
    frame #21: 0x0000000100a222a2 node`Builtins_InterpreterEntryTrampoline + 194
    frame #22: 0x0000000100a4ee70 node`Builtins_AsyncFunctionAwaitResolveClosure + 48
    frame #23: 0x0000000100acb12e node`Builtins_PromiseFulfillReactionJob + 46
    frame #24: 0x0000000100a41f9a node`Builtins_RunMicrotasks + 602
    frame #25: 0x0000000100a1ff18 node`Builtins_JSRunMicrotasksEntry + 120
    frame #26: 0x0000000100334d79 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2761
    frame #27: 0x0000000100335353 node`v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 83
    frame #28: 0x0000000100335431 node`v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) + 81
    frame #29: 0x0000000100358013 node`v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 403
    frame #30: 0x0000000100357e51 node`v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) + 33
    frame #31: 0x0000000100267568 node`v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) + 616
    frame #32: 0x0000000100266afc node`v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) + 524
    frame #33: 0x0000000100266222 node`v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) + 258
    frame #34: 0x0000000100a894b9 node`Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit + 57
    frame #35: 0x0000000100a222a2 node`Builtins_InterpreterEntryTrampoline + 194
    frame #36: 0x0000000100a1ffba node`Builtins_JSEntryTrampoline + 90
    frame #37: 0x0000000100a1fd98 node`Builtins_JSEntry + 120
    frame #38: 0x0000000100334e98 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 3048
    frame #39: 0x00000001003342a2 node`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 210
    frame #40: 0x0000000100212b23 node`v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 435
    frame #41: 0x0000000100001c24 node`node::InternalCallbackScope::Close() + 420
    frame #42: 0x000000010000226a node`node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) + 730
    frame #43: 0x00000001000163f8 node`node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) + 200
    frame #44: 0x0000000100159bcd node`node::(anonymous namespace)::CompressionStream<node::(anonymous namespace)::ZlibContext>::AfterThreadPoolWork(int) + 189
    frame #45: 0x00000001009ff368 node`uv__work_done + 184
    frame #46: 0x0000000100a04433 node`uv__async_io + 355
    frame #47: 0x0000000100a180ac node`uv__io_poll + 2060
    frame #48: 0x0000000100a049a1 node`uv_run + 401
    frame #49: 0x00000001000f2075 node`node::NodeMainInstance::Run() + 309
    frame #50: 0x00000001000851b6 node`node::Start(int, char**) + 294
    frame #51: 0x00007fff203e3f5d libdyld.dylib`start + 1

@jaysoffian
Copy link
Author

Using node --heap-prof --track-heap-objects test.js also crashes without even attaching DevTools:

Process 52825 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffbffffffff)
    frame #0: 0x0000000100680844 node`v8::internal::AllocationTracker::AllocationEvent(unsigned long, int) + 164
node`v8::internal::AllocationTracker::AllocationEvent:
->  0x100680844 <+164>: movq   -0x1(%r15), %rsi
    0x100680848 <+168>: decq   %r15
    0x10068084b <+171>: leaq   -0x48(%rbp), %rdi
    0x10068084f <+175>: callq  0x1005dc350               ; v8::internal::HeapObject::SizeFromMap(v8::internal::Map) const
Target 0: (node) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffbffffffff)
  * frame #0: 0x0000000100680844 node`v8::internal::AllocationTracker::AllocationEvent(unsigned long, int) + 164
    frame #1: 0x000000010037c82e node`v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 574
    frame #2: 0x00000001003ae7c1 node`v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 33
    frame #3: 0x00000001003ae861 node`v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) + 33
    frame #4: 0x0000000100375778 node`v8::internal::FactoryBase<v8::internal::Factory>::NewByteArray(int, v8::internal::AllocationType) + 56
    frame #5: 0x0000000100318b02 node`v8::internal::TranslatedState::EnsureJSObjectAllocated(v8::internal::TranslatedValue*, v8::internal::Handle<v8::internal::Map>) + 114
    frame #6: 0x00000001003181cb node`v8::internal::TranslatedState::EnsureCapturedObjectAllocatedAt(int, std::__1::stack<int, std::__1::deque<int, std::__1::allocator<int> > >*) + 811
    frame #7: 0x0000000100313b76 node`v8::internal::TranslatedState::EnsureObjectAllocatedAt(v8::internal::TranslatedValue*) + 374
    frame #8: 0x0000000100310b7e node`v8::internal::TranslatedValue::GetValue() + 238
    frame #9: 0x0000000100310919 node`v8::internal::Deoptimizer::MaterializeHeapObjects() + 265
    frame #10: 0x00000001006e44fe node`v8::internal::Runtime_NotifyDeoptimized(int, unsigned long*, v8::internal::Isolate*) + 430
    frame #11: 0x0000000100a893d9 node`Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit + 57
    frame #12: 0x0000000100a22ef4 node`Builtins_NotifyDeoptimized + 20
    frame #13: 0x0000000100a228a0 node`Builtins_InterpreterEnterBytecodeAdvance + 224
    frame #14: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #15: 0x0000000100a227c0 node`Builtins_InterpreterPushArgsThenConstructWithFinalSpread + 96
    frame #16: 0x0000000100a222a2 node`Builtins_InterpreterEntryTrampoline + 194
    frame #17: 0x0000000100a4ee70 node`Builtins_AsyncFunctionAwaitResolveClosure + 48
    frame #18: 0x0000000100acb12e node`Builtins_PromiseFulfillReactionJob + 46
    frame #19: 0x0000000100a41f9a node`Builtins_RunMicrotasks + 602
    frame #20: 0x0000000100a1ff18 node`Builtins_JSRunMicrotasksEntry + 120
    frame #21: 0x0000000100334d79 node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2761
    frame #22: 0x0000000100335353 node`v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 83
    frame #23: 0x0000000100335431 node`v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) + 81
    frame #24: 0x0000000100358013 node`v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 403
    frame #25: 0x0000000100357e51 node`v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) + 33
    frame #26: 0x0000000100001b5a node`node::InternalCallbackScope::Close() + 218
    frame #27: 0x00000001000015de node`node::InternalCallbackScope::~InternalCallbackScope() + 14
    frame #28: 0x0000000100061e18 node`node::Environment::RunTimers(uv_timer_s*) + 568
    frame #29: 0x0000000100a000a7 node`uv__run_timers + 103
    frame #30: 0x0000000100a048dd node`uv_run + 205
    frame #31: 0x00000001000f2075 node`node::NodeMainInstance::Run() + 309
    frame #32: 0x00000001000851b6 node`node::Start(int, char**) + 294
    frame #33: 0x00007fff203e3f5d libdyld.dylib`start + 1
(lldb)

@jaysoffian
Copy link
Author

(The crash in AllocationTracker::AllocationEvent, which I realize is a different backtrace than the one I first reported, also occurs under node v16.3.0).

@lukas-bluescape lukas-bluescape mentioned this issue Jul 8, 2021
Closed
@lukas-bluescape
Copy link

lukas-bluescape commented Jul 8, 2021 •
edited
Loading

I seem to have managed to reproduce the same heap snapshot crash, with my own debug build from commit ce4d224 on the v14.x branch (EDIT: I was unable to reproduce this crash from my build at commit 1543497 on the v16.x branch; it appears to be fixed there). This appears to have provided me with some more details from the core file, including a few additional stack frames on top of ExtractContextReferences, which I'm hoping might be more helpful to somebody who knows their way around the codebase better:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000561128b72320 in v8::internal::heap_internals::MemoryChunk::GetHeap (this=0x100000000)
    at ../deps/v8/src/heap/heap-write-barrier-inl.h:82
82	    Heap* heap = *reinterpret_cast<Heap**>(reinterpret_cast<Address>(this) +
[Current thread is 1 (Thread 0x7faf295e2780 (LWP 30))]
(gdb) bt
#0  0x0000561128b72320 in v8::internal::heap_internals::MemoryChunk::GetHeap (this=0x100000000)
    at ../deps/v8/src/heap/heap-write-barrier-inl.h:82
#1  v8::internal::GetHeapFromWritableObject (object=...) at ../deps/v8/src/execution/isolate-utils-inl.h:38
#2  v8::internal::ReadOnlyHeap::GetReadOnlyRoots (object=...) at ../deps/v8/src/heap/read-only-heap-inl.h:25
#3  0x00005611293aa293 in v8::internal::HeapObject::GetReadOnlyRoots (this=<optimized out>)
    at ../deps/v8/src/objects/objects-inl.h:691
#4  v8::internal::HeapObject::IsUndefined (this=<optimized out>) at ../deps/v8/src/objects/objects-inl.h:117
#5  v8::internal::Context::has_extension (this=0x7fff4f67a3e8) at ../deps/v8/src/objects/contexts-inl.h:99
#6  v8::internal::Context::has_extension (this=0x7fff4f67a3e8) at ../deps/v8/src/objects/contexts-inl.h:98
#7  v8::internal::V8HeapExplorer::ExtractContextReferences (this=this@entry=0x7fff4f67a6a8, entry=entry@entry=0x561134c04738, 
    context=...) at ../deps/v8/src/profiler/heap-snapshot-generator.cc:1000
#8  0x00005611293ab5e8 in v8::internal::V8HeapExplorer::ExtractReferences (this=this@entry=0x7fff4f67a6a8, 
    entry=entry@entry=0x561134c04738, obj=...) at ../deps/v8/src/objects/contexts-inl.h:48
#9  0x00005611293aba92 in v8::internal::V8HeapExplorer::IterateAndExtractReferences (this=this@entry=0x7fff4f67a6a8, 
    generator=generator@entry=0x7fff4f67a690) at ../deps/v8/src/profiler/heap-snapshot-generator.cc:1516
#10 0x00005611293ace2f in v8::internal::HeapSnapshotGenerator::FillReferences (this=0x7fff4f67a690)
    at ../deps/v8/src/profiler/heap-snapshot-generator.cc:2094
#11 v8::internal::HeapSnapshotGenerator::GenerateSnapshot (this=this@entry=0x7fff4f67a690)
    at ../deps/v8/src/profiler/heap-snapshot-generator.cc:2058
#12 0x00005611293952f0 in v8::internal::HeapProfiler::TakeSnapshot (this=0x56112e0ab480, control=0x0, resolver=0x0, 
    treat_global_objects_as_roots=<optimized out>) at ../deps/v8/src/profiler/heap-profiler.cc:75
#13 0x00005611288628ec in node::heap::CreateHeapSnapshotStream (args=...) at ../src/heap_utils.cc:360
#14 0x0000561128c4ca85 in v8::internal::FunctionCallbackArguments::Call (this=this@entry=0x7fff4f67aa10, handler=..., 
    handler@entry=...) at ../deps/v8/src/api/api-arguments-inl.h:158
#15 0x0000561128c4daa0 in v8::internal::(anonymous namespace)::HandleApiCallHelper<false> (
    isolate=isolate@entry=0x56112e08eb40, function=..., function@entry=..., new_target=..., new_target@entry=..., 
    fun_data=..., receiver=..., receiver@entry=..., args=...) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#16 0x0000561128c51fd7 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=isolate@entry=0x56112e08eb40)
    at ../deps/v8/src/handles/handles.h:137
#17 0x0000561128c52e70 in v8::internal::Builtin_HandleApiCall (args_length=5, args_object=0x7fff4f67abb8, 
    isolate=0x56112e08eb40) at ../deps/v8/src/builtins/builtins-api.cc:129
#18 0x0000561129b118e0 in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit ()
    at ../../deps/v8/../../deps/v8/src/builtins/promise-misc.tq:111
#19 0x0000561129910ae2 in Builtins_InterpreterEntryTrampoline () at ../../deps/v8/../../deps/v8/src/builtins/convert.tq:16
#20 0x00000002707c0471 in ?? ()
#21 0x000004d91c772321 in ?? ()

@GioMac
Copy link

GioMac commented Aug 30, 2021

same issue with 14.17.3-2.module+el8.4.0+639+18660d0d (rocky 8)
cannot reproduce with v16
workaround: stopped inbound traffic to app (not sure if any other parallel tasks affect behavior)

@Venryx
Copy link

Venryx commented Jan 15, 2022

Your issues may be similar to the one I hit: #41539

Btw @jaysoffian, thank you so much for this small comment:

Using node --heap-prof --track-heap-objects test.js also crashes without even attaching DevTools:

It is what helped me reproduce the problem in a reliable manner, ultimately leading to the small and self-contained repro code linked in the issue above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
inspector Issues and PRs related to the V8 inspector protocol v8 module Issues and PRs related to the "v8" subsystem.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jaysoffian @GioMac @joyeecheung @Venryx @Ayase-252 @bl-ue @lukas-bluescape