cross-restart task timing

[hjoliver]

I get frequent questions about why the task timing information shown in the GUI text tree view is lost (temporarily) after a restart. It's because the timings are estimated as a mean of completion times (for each task) over the course of the current run, so it doesn't appear until at least one instance (for each task) has succeeded.

We should store the mean completion time for each task in the run DB, and continue to update it as normal after a restart. May want to use a weighted mean that strongly favours recent instances though, to better handle changes in task run length that could occur at reload or restart?

[matthewrmshin]

We can probably re-calculate this on restart, based on the run times of recent succeeded jobs of each task, which are already stored in the database (as time_run_exit - time_run in the task_jobs table). We may even be able to use the avg function here.

[matthewrmshin]

I have just noticed the method cylc.taskdef.TaskDef.update_mean_total_elapsed_time, which has this statement.

self.elapsed_times.append(t_succeeded - t_started)

However, it appears that we never remove anything from self.elapsed_times, so for a long running suite, it looks like the list will grow forever!

[arjclark]

However, it appears that we never remove anything from self.elapsed_times, so for a long running suite, it looks like the list will grow forever!

Probably best to keep only, say, the last 10

[hjoliver]

Doh! 😖 Yeah, a rolling average of 10 or so would be fine.