Add a configuration to enable a debug faulthandler (PP-1717)

[jonathangreen]

Description

Adds a new env variable PALACE_LOG_DEBUG_TRACEBACK_INTERVAL. This option is disabled when it is zero (the default). When it is non-zero it uses the python faulthandler module to print stack traces of every thread to stderr every INTERVAL seconds.

Motivation and Context

This is useful to figure out the state of the app when a thread is deadlocked, watching the stack traces after the deadlock occurs gives an idea of what is happening. I though this would be useful to add to the main codebase as it might help assist if / when we get deadlocks happening in environments that are not easily reproduced. This could be enabled with a env var to get a bit of additional information to assist with debugging.

How Has This Been Tested?

• Tested locally, I was using it to help track down a deadlock in PG.

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[tdilauro]

🪵👀

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.79%. Comparing base (fee4a3d) to head (25bce2c).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2065      +/-   ##
==========================================
+ Coverage   90.70%   90.79%   +0.08%     
==========================================
  Files         343      344       +1     
  Lines       40584    40598      +14     
  Branches     6588     8794    +2206     
==========================================
+ Hits        36813    36859      +46     
+ Misses       2501     2484      -17     
+ Partials     1270     1255      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.