When is this a problem in the real world?

When your app does something memory-intensive but not very often, so you lower accepts value to release memory back to the OS. AFAIK this (or memory leakage) is the primary use case for similar setting in other web servers, e. g. httpd's MaxConnectionsPerChild.

In my case setting accepts to 10 resulted in several dozens core dumps a day on a modest 2 RPS.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

The problem seems too obscure for us to fix in Mojolicious.

We see 100s of similar core dumps a week in production with accepts set to 100 and requests set to 70. Although it isn't a significant issue, it does cause many spurious logs that must be filtered through when monitoring for real issues.

We see the same issue in our production. At this point, our conclusion is that when exit is called, Perl will reset the signal handlers early in the process before it actually exits. This appears to happen even before the END blocks are executed. So if a worker is in the process of shutting down and is executing an END block, when the parent process sends it the QUIT signal, the worker process will core dump.

That Perl resets the signal handlers before END is executed is illustrated by the following short script:

$SIG{QUIT} = sub {};
kill 'QUIT', $$;
print "After first quit\n";

END {
    kill 'QUIT', $$;
    print "After second quit\n";
}

Running this program will only cause the first print-statement to be executed, as the program will core-dump before reaching the second statement (tested on perl 5.34.0)

It seems that it is only handlers that are reset. If sub {} is switched out with 'IGNORE', the print-statement in the END block will also be executed.

I'm not familiar with the innards of the perl-interpreter, but it seems plausible that it will have to reset the signal handlers before exiting, leaving a small window open for a race between exiting and receiving a QUIT signal from its parent.

localizing the signal handler in the END block results in handling the second kill.

I can't see that localizing the signal handler in the END block eliminates the race completely. We can see that Perl resets the signal handler before the END blocks are executed. The QUIT signal could come after the signal handlers are reset and before our END block that reinstalls the signal handler is executed.

AFAICT the potential fix with the before_server_start-hook in the original post eliminates the race. A "simpler" solution, would be to add:

$SIG{QUIT} = 'IGNORE';

inbetween $loop->start; and exit 0; in Mojo::Server::Prefork::_spawn (we are already exiting, so no need to know that we should do it gracefully - that won't have any effect anyway).

I would submit a pull-request for this, but I haven't yet wrapped my head around how to write a test to ensure that it actually works.

@kraih Please consider reopening. We have accepts at 1000 but this issue actually caused an outage for us when Ubuntu started storing core dumps in an unexpected location after a routine apt upgrade.

Here is a maybe less obscure repro which dumps core for me about 50% of the time:

Server (using strace in case core dumps are disabled):

strace -e kill perl -Mojo -E 'get "/" => {inline => "%= sleep 5"}; app->start' prefork -a 1

Client:

while sleep 1; do curl http://127.0.0.1:3000/; done

The trick is doing enough work (sleep 5) that a heartbeat happens while the IO loop is shutting down after reaching its maximum number of requests (i.e. max_accepts). That can end up trying to signal SIGQUIT when the worker has unregistered its handler during exit().

It turns out the Prefork signal handlers are local so an effective workaround is to globally ignore SIGQUIT before startup. As @ksmadsen has shown, this will also ignore the signals during exit().

My test server then becomes:

strace -e kill perl -Mojo -E '$SIG{QUIT} = "IGNORE"; get "/" => {inline => "%= sleep 5"}; app->start' prefork -a 1

which never core dumps for me.

I still think a change is warranted so that all users do not have to know about such a workaround.

This bit us hard. Eating up huge amounts of disk space once we lowered accepts.

There's a possible fix, unfortunately nobody has reviewed it yet.

FWIW, this simple workaround has completely solved the problem for us (this is also what I discussed above):

BEGIN {
    # Protect against stray signals from the process manager
    $SIG{QUIT} = 'IGNORE';
}

The PR #1883 ensures that the stray signals never happen and the workaround is no longer needed.