[Bug]: The server stopped responding to certain requests after starting to terminate an instance

[peterschmidt85]

No certain steps to reproduce.

Once, when terminating an instance, the server stopped responding to certain requests.

The last server logs:

{
    "message": "run(22afe2)curly-robin-1: run status has changed PROVISIONING -> TERMINATING",
    "logger": "dstack._internal.server.background.tasks.process_runs",
    "timestamp": "2024-05-17 13:47:13,146",
    "level": "INFO"
}

After that, there were no server logs despite other requests to the server.

The run got stuck in the terminating state.

The command dstack stop -x <run-name> hanged.

Other requests such as dstack ps worked as usual.

After a restart of the server, the problem has gone.