Crash when running a full recostruction workflow with LST used for track building

[VourMa]

During the 16th Patatrack hackathon, I tried a full reconstruction workflow, instead of the tracking-only that we currently run for validation, using LST for tracking building (--procModifier trackingLST). The cmsRun command crash with the segmentation violation quoted below. Using --procModifier trackingIters01 doesn't make a difference.

Not something for now but documenting it so that we don't forget.

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Jul 31 03:16:22 PDT 2024
Thread 11 (Thread 0x7ff352e00700 (LWP 3997266) "cmsRun"):
#0  0x00007ff4f9132ac1 in poll () from /lib64/libc.so.6
#1  0x00007ff4f551b11f in full_read.constprop () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007ff4f54cf5dc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00007ff4f54cff40 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007ff490bac3f8 in GsfTrajectoryFitter::fitOne(TrajectorySeed const&, std::vector<std::shared_ptr<TrackingRecHit const>, std::allocator<std::shared_ptr<TrackingRecHit const> > > const&, TrajectoryStateOnSurface const&, TrajectoryFitter::fitType) const () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libTrackingToolsGsfTracking.so
#6  0x00007ff40893f820 in GoodSeedProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginRecoParticleFlowPFTrackingPlugins.so
#7  0x00007ff4fc64ad45 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#8  0x00007ff4fc62f76e in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x00007ff4fc5badd9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x00007ff4fc5bd518 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ff4fc8a7f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#12 0x00007ff4fc09795b in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter> (t=0x7ff3b8a87a00, waiter=..., this=0x7ff4f75b3e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#13 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter> (t=0x0, waiter=..., this=0x7ff4f75b3e80) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#14 tbb::detail::r1::arena::process (tls=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/arena.cpp:137
#15 tbb::detail::r1::market::process (this=<optimized out>, j=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/market.cpp:599
#16 0x00007ff4fc099b0e in tbb::detail::r1::rml::private_worker::run (this=0x7ff4f9ecf100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/private_server.cpp:271
#17 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7ff4f9ecf100) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#18 0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#19 0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7ff353c00700 (LWP 3997262) "cmsRun"):
#0  0x00007ff4f940e48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff466d39bf0 in alpaka::core::CallbackThread::startWorkerThread()::{lambda()#1}::operator()() const () from /home/users/evourlio/LSTinCMSSW/movingFullyToCMSSW/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libRecoTrackerLSTCoreCudaAsync.so
#2  0x00007ff4f9cd8a73 in std::execute_native_thread_routine (__p=0x7ff41d805030) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#3  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#4  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7ff41e400700 (LWP 3997261) "cmsRun"):
#0  0x00007ff4f9410dd6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007ff4f9410ec8 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007ff4e1c2c242 in ?? () from /lib64/libcuda.so.1
#3  0x00007ff4e1c3c233 in ?? () from /lib64/libcuda.so.1
#4  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7ff41f200700 (LWP 3955806) "cmsRun"):
#0  0x00007ff4f940e48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff47baf171e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007ff47baf1cd3 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007ff47baef448 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007ff46ae83a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7ff420000700 (LWP 3955805) "cmsRun"):
#0  0x00007ff4f940e48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff47baf171e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007ff47baf1cd3 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007ff47baef448 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007ff46ae83a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7ff420e00700 (LWP 3955804) "cmsRun"):
#0  0x00007ff4f940e48c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff47baf171e in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#2  0x00007ff47baf1cd3 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#3  0x00007ff47baef448 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_cc.so.2
#4  0x00007ff46ae83a02 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/external/el8_amd64_gcc12/lib/libtensorflow_framework.so.2
#5  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7ff4ab600700 (LWP 3955742) "cuda-EvtHandlr"):
#0  0x00007ff4f9132ac1 in poll () from /lib64/libc.so.6
#1  0x00007ff4e1c4164f in ?? () from /lib64/libcuda.so.1
#2  0x00007ff4e1d0f18f in ?? () from /lib64/libcuda.so.1
#3  0x00007ff4e1c3c233 in ?? () from /lib64/libcuda.so.1
#4  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7ff4c0a00700 (LWP 3955741) "cuda-EvtHandlr"):
#0  0x00007ff4f9132ac1 in poll () from /lib64/libc.so.6
#1  0x00007ff4e1c4164f in ?? () from /lib64/libcuda.so.1
#2  0x00007ff4e1d0f18f in ?? () from /lib64/libcuda.so.1
#3  0x00007ff4e1c3c233 in ?? () from /lib64/libcuda.so.1
#4  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7ff4c7800700 (LWP 3955716) "cuda0000300000d"):
#0  0x00007ff4f9132ac1 in poll () from /lib64/libc.so.6
#1  0x00007ff4e1c4164f in ?? () from /lib64/libcuda.so.1
#2  0x00007ff4e1d0f18f in ?? () from /lib64/libcuda.so.1
#3  0x00007ff4e1c3c233 in ?? () from /lib64/libcuda.so.1
#4  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7ff4c8200700 (LWP 3955708) "cmsRun"):
#0  0x00007ff4f94126a2 in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff4f54cc637 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x00007ff4f54cf50a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x00007ff4f9cd8a73 in std::execute_native_thread_routine (__p=0x7ff4ed1bea80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00007ff4f94081ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4f90398d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff4fb3fe640 (LWP 3932427) "cmsRun"):
#0  0x00007ff4f9108098 in nanosleep () from /lib64/libc.so.6
#1  0x00007ff4f9107f9e in sleep () from /lib64/libc.so.6
#2  0x00007ff4f54cc4e0 in sig_pause_for_stacktrace () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff422d87b81 in std::__detail::_Map_base<DetId, std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > >, std::allocator<std::pair<DetId const, std::vector<std::pair<int, float>, std::allocator<std::pair<int, float> > > > >, std::__detail::_Select1st, std::equal_to<DetId>, std::hash<DetId>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](DetId const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#5  0x00007ff422d8bcd0 in TSToSimTSHitLCAssociatorByEnergyScoreImpl::makeConnections(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#6  0x00007ff422d8ddee in TSToSimTSHitLCAssociatorByEnergyScoreImpl::associateSimToReco(edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&, edm::Handle<std::vector<reco::CaloCluster, std::allocator<reco::CaloCluster> > > const&, edm::Handle<std::vector<SimCluster, std::allocator<SimCluster> > > const&, edm::Handle<std::vector<CaloParticle, std::allocator<CaloParticle> > > const&, edm::Handle<std::vector<ticl::Trackster, std::allocator<ticl::Trackster> > > const&) const () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#7  0x00007ff422d89e77 in TSToSimTSHitLCAssociatorEDProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/pluginSimCalorimetryHGCalAssociatorProducersAuto.so
#8  0x00007ff4fc636994 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#9  0x00007ff4fc6298fe in edm::WorkerT<edm::global::EDProducerBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#10 0x00007ff4fc5badd9 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#11 0x00007ff4fc5bd518 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#12 0x00007ff4fc8a7f28 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreConcurrency.so
#13 0x00007ff4fc0a0281 in tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x7ff4f75b3e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:322
#14 tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter> (waiter=..., t=<optimized out>, this=0x7ff4f75b3e00) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.h:458
#15 tbb::detail::r1::task_dispatcher::execute_and_wait (t=<optimized out>, wait_ctx=..., w_ctx=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/task_dispatcher.cpp:168
#16 0x00007ff4fc53e79b in edm::FinalWaitingTask::wait() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#17 0x00007ff4fc54e03a in edm::EventProcessor::processRuns() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#18 0x00007ff4fc547f41 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre3/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x00000000004074ef in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const ()
#20 0x00007ff4fc08c9ad in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_1_0_pre3-el8_amd64_gcc12/build/CMSSW_14_1_0_pre3-build/BUILD/el8_amd64_gcc12/external/tbb/v2021.9.0-5849be8e21b090e14f1b189539cee138/tbb-v2021.9.0/src/tbb/arena.cpp:688
#21 0x0000000000408ed2 in main::{lambda()#1}::operator()() const ()
#22 0x000000000040517c in main ()

Current Modules:

Module: GoodSeedProducer:trackerDrivenElectronSeeds (crashed)
Module: TSToSimTSHitLCAssociatorEDProducer:tracksterSimTracksterAssociationPRbyCLUE3D

A fatal system signal has occurred: segmentation violation
Segmentation fault (core dumped)

[slava77]

please recompile TrackingTools/GsfTracking with debug info (scram with USER_CXXFLAGS=-g) and rerun to get more details what's happening; perhaps even check details with gdb if they are usable