Additional information on unet baseline

[HugoSchmutz]

How long to train a Unet on 800 epochs with your hyperparameters?
How many samples per volume do you use at each epoch?
Do you use a scheduler?

Best,
Hugo

[thomaskuestner]

You can find all information on the baseline nnUNet model in our publication:
https://www.nature.com/articles/s41597-022-01718-3#Sec11
If not stated otherwise in the manuscript, the default nnUNet parameters were chosen.

[HugoSchmutz]

Hi
thank you for your answer. My question was not clear, I was wondering how long in term of time does it take to train the model, approximatively?
I am trying to replicate the results but my current version of code is terribly long. (~30mn per epoch)

[thomaskuestner]

Thanks for clarifying. Well, nnUNet can get very slow if it falls back to a single-threaded process (with all other workers sleeping instead of actually prefetching the data) or if it is not run on the GPU. Are you loading data from a local hard drive or via a network drive which could bottleneck you?
Did you check for this?
You can also refer to the documentation here for more info: https://github.com/MIC-DKFZ/nnUNet/blob/nnunetv1/documentation/common_problems_and_solutions.md

Usually for us an epoch took around a few minutes (<5 min).

[HugoSchmutz]

Thanks for your answer. I am actually training a Unet not a nnUnet. I am using torchio to manage the patch sampling.
For now, I am running it on a single GPU on a server. The data is on the server too.

The training is actually quite fast, it is the loading of images, sampling of patches and preparation of dataloaders which is long. Any advice on how to faster this part?

[thomaskuestner]

In this regard the only advice is to run multiple workers for pre-fetching and pre-processing the data. All tweaks depend on your hardware and actual setup.

[HugoSchmutz]

Thank you for your advices. Indeed, multi-workers help

[thomaskuestner]

You are welcome.