Increasing stack size limit results in inconsistent behavior, stack overflow

[ccope]

I'm writing some toy programs to learn rust and encountered some strange behavior when changing stack size limits.

I tried building and running this code: https://github.com/ccope/oom-rs

I expected to see this happen: Printing out the changed stack size limit, followed by the test strings.

Instead, this happened:

• 1.19.0:

  • debug: Failed

    $ ./target/debug/oom 
    the narwhal bacons at midnight
    Old value: 8388608
    New value: 32768000
    Segmentation fault (core dumped)
    

    • strace: https://gist.github.com/ccope/88b5d1374ebdeaf06af0a2eec25ae43c
  • release: Failed

    $ ./target/release/oom 
    Segmentation fault (core dumped)
    

    • strace: https://gist.github.com/ccope/94a231440ac6654cdfbbb5c90377164a

• 1.21.0-nightly:

  • debug: Pass

    $ ./target/debug/oom
    the narwhal bacons at midnight
    Old value: 8388608
    New value: 32768000
    Approached the castle
    First shalt thou take out the Holy Pin, then shalt thou count to 3, no more, no less.
    <snip>
    

  • release: Fail

    $ ./target/release/oom 
    
    thread 'main' has overflowed its stack
    fatal runtime error: stack overflow
    Aborted (core dumped)
    

    • strace: https://gist.github.com/ccope/12b3ffb98d5e987603f1a072144364ae

Meta

Ubuntu 16.04, kernel 4.10.0-32-generic

rustc --version --verbose:

1.19.0:

rustc 1.19.0 (0ade33941 2017-07-17)
binary: rustc
commit-hash: 0ade339411587887bf01bcfa2e9ae4414c8900d4
commit-date: 2017-07-17
host: x86_64-unknown-linux-gnu
release: 1.19.0
LLVM version: 4.0

1.21.0:

rustc 1.21.0-nightly (df511d554 2017-08-14)
binary: rustc
commit-hash: df511d5548ebb7f971abcd3a5283cb1d37b64596
commit-date: 2017-08-14
host: x86_64-unknown-linux-gnu
release: 1.21.0-nightly
LLVM version: 4.0

[ccope]

Fired up GDB, and the segfault happened in compiler_builtins::probestack::__rust_probestack () at /checkout/src/rustc/compiler_builtins_shim/../../libcompiler_builtins/src/probestack.rs:55 (in the nightly rust build). Then I set a breakpoint on that function and was able to get a backtrace that led me to std::rt::lang_start() and sys::thread::guard::init(). Looking at this code, I wouldn't expect setrlimit(RLIMIT_STACK) to ever work inside main() because the guard pages will already be present in the middle of the address space.

[Mark-Simulacrum]

cc @alexcrichton

[alexcrichton]

I believe this is all working as intended so I'm going to close this, but thanks for the report!

In 1.20.0 Rust gained support for "stack probes" which is the __rust_probestack function that you see. A stack probe is basically just a function prologue which touches each page of a stack if the stack size is greater than a page (which it is in this case an the stack-allocated bfa array. Note that with the addition of these stack probes it also necessitated the removal of Rust's main-thread guard page on Linux due to a variety of reasons, the tl;dr; of which being that it's already taken care of.

So what's happening here is:

• 1.19 debug mode - There's no stack probes. The main thread has a Rust-allocated guard page, however. I believe what's happening is that you're hitting this guard page (naturally w/ the huge array) and the runtime is misdiagnosing the segfault, not detecting a stack overflow and printing out something nice.
• 1.19 release mode - The huge_tracts_of_land function probably gets inlined into the main function, which makes it hit the guard page and unallocated memory even faster. What's probaby happening is the main function completely jumps over the guard page, and because the stack is too big when you're calling println-related functions it's just hitting random unmapped memory below the guard page.
• 1.21 debug mode - there's stack probes enabled here but notably also there's no main thread guard page allocated by Rust. This means that everything works as expected as Rust touches the stack pages and finds that the setrlimit made room for Linux to allocate all of them.
• 1.21 release mode - like in 1.19 the huge_tracts_of_land function is likely inlined, making the main stack way bigger than it should be, triggering a segfault in the probestack function as it touches each page of the stack (and eventually hitting an unmapped page). In this case the runtime correctly diagnoses the segfault as a stack overflow.

In any case I don't think there's anything more to do here. You can get the example working in debug/release mode on 1.20+ with the #[inline(never)] attribute on the huge_tracts_of_land function.