Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent segmentation fault while running CFITSIO tests on nightly #54724

Open
jishnub opened this issue Jun 7, 2024 · 4 comments
Open

Intermittent segmentation fault while running CFITSIO tests on nightly #54724

jishnub opened this issue Jun 7, 2024 · 4 comments
Labels
kind:heisenbug This bug occurs unpredictably kind:rr trace included
Milestone
1.12

Comments

@jishnub
Copy link
Contributor

jishnub commented Jun 7, 2024

I am using the PR JuliaAstro/CFITSIO.jl#23 to run the tests, which should be somewhat safer than the master.

An example of a stacktrace for a failure:

[22051] signal 11 (1): Segmentation fault
in expression starting at /home/jishnu/Dropbox/JuliaPackages/CFITSIO.jl/test/runtests.jl:117
jl_gc_pool_alloc_inner at /cache/build/tester-amdci4-12/julialang/julia-master/src/gc.c:1335
jl_gc_pool_alloc_noinline at /cache/build/tester-amdci4-12/julialang/julia-master/src/gc.c:1392 [inlined]
jl_gc_alloc_ at /cache/build/tester-amdci4-12/julialang/julia-master/src/julia_internal.h:507 [inlined]
jl_gc_alloc at /cache/build/tester-amdci4-12/julialang/julia-master/src/gc.c:3990
_new_genericmemory_ at /cache/build/tester-amdci4-12/julialang/julia-master/src/genericmemory.c:56 [inlined]
jl_alloc_genericmemory at /cache/build/tester-amdci4-12/julialang/julia-master/src/genericmemory.c:99
GenericMemory at ./boot.jl:531 [inlined]
Array at ./boot.jl:593 [inlined]
fits_read_key_str at /home/jishnu/Dropbox/JuliaPackages/CFITSIO.jl/src/CFITSIO.jl:526
readhealpix at /home/jishnu/Dropbox/JuliaPackages/CFITSIO.jl/test/runtests.jl:103
macro expansion at /cache/build/tester-amdci4-12/julialang/julia-master/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:676 [inlined]
#4 at /home/jishnu/Dropbox/JuliaPackages/CFITSIO.jl/test/runtests.jl:200
#mktempdir#28 at ./file.jl:835
mktempdir at ./file.jl:831
unknown function (ip: 0x7fc788f545e6)
mktempdir at ./file.jl:831
unknown function (ip: 0x7fc788f54552)
jl_apply at /cache/build/tester-amdci4-12/julialang/julia-master/src/julia.h:2190 [inlined]
do_call at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:127
eval_value at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:224
eval_stmt_value at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:175 [inlined]
eval_body at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:662
eval_body at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:542
eval_body at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:542
jl_interpret_toplevel_thunk at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:873
jl_toplevel_eval_flex at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:969
jl_toplevel_eval_flex at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:909
ijl_toplevel_eval at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:980
ijl_toplevel_eval_in at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:1022
eval at ./boot.jl:432 [inlined]
include_string at ./loading.jl:2589
_include at ./loading.jl:2649
include at ./sysimg.jl:38
unknown function (ip: 0x7fc788f00082)
jl_apply at /cache/build/tester-amdci4-12/julialang/julia-master/src/julia.h:2190 [inlined]
do_call at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:127
eval_value at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:224
eval_stmt_value at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:175 [inlined]
eval_body at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:662
jl_interpret_toplevel_thunk at /cache/build/tester-amdci4-12/julialang/julia-master/src/interpreter.c:873
jl_toplevel_eval_flex at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:969
jl_toplevel_eval_flex at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:909
ijl_toplevel_eval at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:980
ijl_toplevel_eval_in at /cache/build/tester-amdci4-12/julialang/julia-master/src/toplevel.c:1022
eval at ./boot.jl:432 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:533
jfptr__start_71469.1 at /home/jishnu/.julia/juliaup/julia-nightly/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci4-12/julialang/julia-master/src/julia.h:2190 [inlined]
true_main at /cache/build/tester-amdci4-12/julialang/julia-master/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/tester-amdci4-12/julialang/julia-master/src/jlapi.c:1059
main at /cache/build/tester-amdci4-12/julialang/julia-master/cli/loader_exe.c:58
unknown function (ip: 0x7fc79ffada8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 8461062 (Pool: 8460879; Big: 183); GC: 7
ERROR: Package CFITSIO errored during testing (received signal: 11)

The rr trace is available at https://julialang-dumps.s3.amazonaws.com/reports/2024-06-07T09-08-04-jishnub.tar.zst

This could be from an unsafe operation in the package, but it would be good to be sure.
@jishnub jishnub added kind:rr trace included kind:heisenbug This bug occurs unpredictably labels Jun 7, 2024
@KristofferC KristofferC added this to the 1.12 milestone Jun 7, 2024
@maleadt
Copy link
Member

maleadt commented Jun 7, 2024 •
edited
Loading

Are you using CFITSIO through FITSIO.jl? That package is known to be buggy/unsafe: JuliaAstro/FITSIO.jl#194, so in that case this is likely to be a dup of #52951.

@jishnub
Copy link
Contributor Author

jishnub commented Jun 8, 2024

No, in this case I was using it directly

@jishnub
Copy link
Contributor Author

jishnub commented Jul 4, 2024

This issue seems to occur on v1.11.0-rc1 as well, as may be seen in the run https://github.com/JuliaAstro/CFITSIO.jl/actions/runs/9797260895/job/27053504169?pr=32:

[3482] signal 11 (1): Segmentation fault: 11
in expression starting at /Users/runner/work/CFITSIO.jl/CFITSIO.jl/test/runtests.jl:117
jl_gc_pool_alloc_inner at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/gc.c:1335 [inlined]
jl_gc_pool_alloc_noinline at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/gc.c:1392 [inlined]
jl_gc_alloc_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/./julia_internal.h:523
_new_genericmemory_ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/genericmemory.c:56
GenericMemory at ./boot.jl:515 [inlined]
Array at ./boot.jl:580 [inlined]
fits_read_key_str at /Users/runner/work/CFITSIO.jl/CFITSIO.jl/src/CFITSIO.jl:529
readhealpix at /Users/runner/work/CFITSIO.jl/CFITSIO.jl/test/runtests.jl:103
macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:676 [inlined]
#4 at /Users/runner/work/CFITSIO.jl/CFITSIO.jl/test/runtests.jl:200
#mktempdir#28 at ./file.jl:819
mktempdir at ./file.jl:815
unknown function (ip: 0x10bd0a67d)
mktempdir at ./file.jl:815
unknown function (ip: 0x10bd0a5b9)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:663
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:539
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:539
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
include_string at ./loading.jl:2543
_include at ./loading.jl:2603
include at ./sysimg.jl:38
unknown function (ip: 0x10bc70079)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
do_call at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_stmt_value at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:174
eval_body at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:429 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:526
jfptr__start_71121 at /Users/runner/hostedtoolcache/julia/1.11.0-rc1/x64/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/./julia.h:2156 [inlined]
true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-grannysmith-C07ZM05NJYVY.0/build/default-grannysmith-C07ZM05NJYVY-0/julialang/julia-release-1-dot-11/src/jlapi.c:1059
Allocations: 8849832 (Pool: 8849512; Big: 320); GC: 8
ERROR: LoadError: Package CFITSIO errored during testing (received signal: 11)
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/Types.jl:71
 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
   @ Pkg.Operations ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2100
 [3] test
   @ ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1985 [inlined]
 [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Vector{String}, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}})
   @ Pkg.API ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/API.jl:475
 [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{coverage::Bool, julia_args::Vector{String}, force_latest_compatible_version::Bool})
   @ Pkg.API ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/API.jl:159
 [6] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::@Kwargs{coverage::Bool, julia_args::Vector{String}, force_latest_compatible_version::Bool})
   @ Pkg.API ~/hostedtoolcache/julia/1.11.0-rc1/x64/share/julia/stdlib/v1.11/Pkg/src/API.jl:174
 [7] top-level scope
   @ ~/work/_actions/julia-actions/julia-runtest/latest/test_harness.jl:15
 [8] include(fname::String)
   @ Main ./sysimg.jl:38
 [9] top-level scope
   @ none:1
in expression starting at /Users/runner/work/_actions/julia-actions/julia-runtest/latest/test_harness.jl:7
Error: Process completed with exit code 1.

@vtjnash
Copy link
Sponsor Member

vtjnash commented Jul 4, 2024

There is a finalizer attached to this struct https://github.com/JuliaAstro/CFITSIO.jl/blob/faa947886570137d53114456fd5b4046a0af7567/src/CFITSIO.jl#L151, but the package fails to GC.@preserve that struct, ever, which could be resulting in the file getting closed unexpectedly in the middle of operations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:heisenbug This bug occurs unpredictably kind:rr trace included
Projects
None yet
Milestone
1.12
Development

No branches or pull requests
4 participants
@vtjnash @maleadt @KristofferC @jishnub