Add multithreading support to ddsim

[wdconinc]

This PR adds multithreading support to ddsim, through the -j flag. More later.

BEGINRELEASENOTES

• Multithreading support in ddsim

ENDRELEASENOTES

[github-actions]

Test Results

 4 files   4 suites   47m 27s ⏱️
24 tests 15 ✅ 0 💤  9 ❌
84 runs  64 ✅ 0 💤 20 ❌

For more details on these failures, see this check.

Results for commit 5bf711c.

[andresailer]

Can you limit to necessary changes only, no gratuitous beautyfications please?
I am not sure if things are re-arranged a lot or diff gets confused.

[wdconinc]

I was trying to :-) The main reason for rearranging is that the worker actions needed to be separated from the master actions, as previously they were intermingled. That required some necessary rearranging.

[wdconinc]

One way to help diff out could be to define helper functions in the order where things are in the file right now, and then use those helper functions in some new code that simply calls them (like __setupWorker). E.g. the UI setup can be put in a function at its current place in the file.

[andresailer]

As is this is still an instance method.

Suggested change

	def __setupWorker(geant4, sim):
	def __setupWorker(self, geant4)

and accordingly below? Or why would this not work?
Then you either need to move out of the class or add the @staticmethod decorator.
And is the return value required by the interface?

[wdconinc]

I can (and should) likely make this work with @staticmethod. The return value appears to be unused. I based the structure on what is done in SiDSim_MT.py, to the extent it provides any prior art.

[wdconinc]

(I was unable to make this work with an instance method since I suspect that by the time __setupWorker is called, the DD4hepSimulation object is not available to the thread that calls __setupWorker.)

[andresailer]

But according to the indentation, this still is an instance method? Just because the first argument isn't self doesn't change that, does it?

[wdconinc]

This is still a work in progress, but let me give some more background here as to the plan (the "more later" from the top).

• The intention is, of course, full backwards compatibility so this runs in the single-threaded G4RunManager with command line options as given now (and therefore must pass all tests unchanged).
• This PR already works in multi-threaded mode for generators like the gun, but
  • Not for external (hepmc) input files: I would like to consider this beyond the scope of this PR since this will require some changes to the input file reader to make sure it does not read the same sequence of events in each file.
  • Adapting to MT has caused single-threaded running to break: As written above, this needs to get fixed, of course.
• There may be changes to the output files that are due to event reordering. These are likely unavoidable.

It is likely that I will be doing some rewrites of the last commit (5bf711c) which is a bit too big for my taste and not as clear as one would like for non-squash merging.