ICE: capacity overflow or "Illegal instruction (core dumped)" with very large static item

[rprichard]

Added by pnkfelix: here is a summary of cases discussed in comment dialogue on issue..

Outcome Matrix	nonzero elem size	zero-sized elems
array length = 0xffff_ffff_ffff_ffff	error: "too big for the current architecture"	takes too long to know outcome?
array length = 0xffff_ffff	LLVM OOM	successful compile after several minutes

Original bug report below

Test case one:

static V1: [u8; 0xffff_ffff] = [0u8; 0xffff_ffff];
fn main() { V1[0]; }

$ rustc test.rs
Illegal instruction (core dumped)

Test case two:

static V1: [u8; 0xffff_ffff_ffff_ffff] = [0u8; 0xffff_ffff_ffff_ffff];
fn main() { V1[0]; }

$ RUST_BACKTRACE=1 rustc test.rs
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
note: run with `RUST_BACKTRACE=1` for a backtrace
thread 'rustc' panicked at 'capacity overflow', /home/rustbuild/src/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/option.rs:330

stack backtrace:
   1:     0x7f4ba1cdfb84 - sys::backtrace::write::h00123aaa25bd3a87l4C
   2:     0x7f4ba1d08528 - panicking::on_panic::h132e5f71ec66ef60YiJ
   3:     0x7f4ba1c3bb0c - rt::unwind::begin_unwind_inner::ha3837a1e391d832bfZI
   4:     0x7f4ba1c3bef1 - rt::unwind::begin_unwind_fmt::h12ca28db54db2d68QXI
   5:     0x7f4ba1d08167 - rust_begin_unwind
   6:     0x7f4ba1d57324 - panicking::panic_fmt::h99de93df94b78c1afXA
   7:     0x7f4ba13e9550 - trans::consts::const_expr_unadjusted::hc13ed55a56b0ff20YAn
   8:     0x7f4ba13ea349 - trans::consts::const_expr::h138812d7f28437d4ron
   9:     0x7f4ba1324a7b - trans::base::get_item_val::h36642395ea7e258dF7t
  10:     0x7f4ba1321799 - trans::base::trans_item::h90869e00e43d4702mCt
  11:     0x7f4ba140a492 - trans::base::trans_crate::h909d471e074343adQyu
  12:     0x7f4ba2338e73 - driver::phase_4_translate_to_llvm::h2d7e84da5a579268mOa
  13:     0x7f4ba2314153 - driver::compile_input::hd802b376244c1b88Rba
  14:     0x7f4ba23cb942 - run_compiler::hc447d2d22477a14av2b
  15:     0x7f4ba23c9713 - thunk::F.Invoke<A, R>::invoke::h3003831543088172262
  16:     0x7f4ba23c8ba9 - rt::unwind::try::try_fn::h6527156377446821573
  17:     0x7f4ba1d7e358 - rust_try_inner
  18:     0x7f4ba1d7e345 - rust_try
  19:     0x7f4ba23c8ecc - thunk::F.Invoke<A, R>::invoke::h1217623451251402195
  20:     0x7f4ba1cf4b18 - sys::thread::create::thread_start::h36340a6690985d3f9TH
  21:     0x7f4b9bbc4181 - start_thread
  22:     0x7f4ba18a347c - __clone
  23:                0x0 - <unknown>

rustc version:

rustc 1.0.0-nightly (e2fa53e59 2015-03-20) (built 2015-03-21)
binary: rustc
commit-hash: e2fa53e593a854a609ae9efe5a1bbe15265f0a6f
commit-date: 2015-03-20
build-date: 2015-03-21
host: x86_64-unknown-linux-gnu
release: 1.0.0-nightly

I didn't see any other open issues mentioning capacity overflow.

[lilyball]

I just hit this. It happens even with arrays whose element type is zero-sized (and therefore the entire array should be zero-sized):

static _FOO: [(); 9223372036854775807] = [(); 9223372036854775807];

[bluss]

This also gives the same ICE:

let _y = &[(); !0];

If you size it just right you actually get an out of memory instead (= abort)

let _z = &[(); 1 << 32];

[steveklabnik]

Triage: tracked by glacier

[steveklabnik]

tests in the OP seem to not build, but no longer ICE.

let _y = &[(); !0]; still ICEs though

[steveklabnik]

This code

static V1: [u8; 0xffff_ffff] = [0u8; 0xffff_ffff];
fn main() { V1[0]; }

no longer ICEs, it does however kick rustc into high gear and take up huge amounts of RAM and time. https://travis-ci.org/steveklabnik/glacier/builds/370159653 is currently attempting to build this and it looks like it's hanging. At the very least, it shouldn't take multiple minutes to compile such a thing.

[pnkfelix]

I spoke to @oli-obk for a bit about this issue. They have some ideas on ways to address this within const-eval. The main question was how to select an actual threshold to act as the limit on memory usage (or maybe individual array length); we agreed that a configurable threshold analogous to the recursion limit would probably make the most sense.

[pnkfelix]

But having said that, I evaluated the four cases of interest that were identified in the comments here, and evaluated their behavior on my x86_64 Linux box.

I saw:

• a controlled exit from rustc,
• a successful compile,
• an LLVM OOM (is the latter considered an ICE in our book? Its a controlled exit, just not controlled by our own Rust code...)
• and I suspect the last case isn't going to finish compiling any time soon. (Update: I'm still waiting after letting it run for 22 hours... And given the numbers involved here, I don't think I or anyone else is going to see that run finish.)

Based on these observations, I'm going to triage this as P-low.

If someone wants to put in the effort to get the const-eval to support a configurable limit on memory usage, great. (@oli-obk even said that it used to have a memory usage limit, and they ripped it out.) But its not something we should prioritize over other problems we face.

[matthiaskrgr]

Seems to no longer ICE with rustc 1.57.0-nightly (497ee321a 2021-09-09)
@rustbot modify labels: E-needs-test

[oli-obk]

The ZST variants now work really fast, it's only the 2^32 variant that takes forever to compile on 64 bit