Implement timeout mechanism

[diegorusso]

Hello,

as discussed on Discourse it would be nice to implement a timeout within pyperfomance.

If a benchmark run hangs, pyperformance doesn’t have a mechanism to time out and terminate the underlying process. We’ve had a few cases where a benchmark was hanging literally forever (deadlock) and the job was terminated by the CI system eventually (after many hours!). This is not ideal because:

• it’s hard to set an appropriate timeout for the whole process as it might vary from machine to machine
• sometimes folks might not have full control of the CI system hence impossible to set a sensible timeout
• having an early failure by pyperformance is preferred than having a timeout at process (pyperformance) level. This allow better usage of CI resources.

Requirements:

• Ideally the timeout would be applied at benchmark run level and not for the whole pyperformance process allowing a more fine grained control in case of timeout.
• If a benchmark times out, it should not fail the whole pyperformance suite but we should mark it as failed (there is already a mechanism to do that).
• There should be able to provide an option at cli level to override the default timeout value.

[diegorusso]

I plan to implement this soon.

[diegorusso]

This is now done in pyperf (psf/pyperf#205) but before merging the PR in pyperformance we need a release in pyperf.