Add metrics + max buffer size to schedule workflow #3872
Conversation
service/worker/scheduler/workflow.go
Outdated
| @@ -673,6 +682,11 @@ func (s *scheduler) resolveOverlapPolicy(overlapPolicy enumspb.ScheduleOverlapPo | |||
|
|
|||
| func (s *scheduler) addStart(nominalTime, actualTime time.Time, overlapPolicy enumspb.ScheduleOverlapPolicy, manual bool) { | |||
| s.logger.Debug("addStart", "nominal", nominalTime, "actual", actualTime, "overlapPolicy", overlapPolicy, "manual", manual) | |||
| if s.tweakables.MaxBufferSize > 0 && len(s.State.BufferedStarts) >= s.tweakables.MaxBufferSize { | |||
| s.logger.Error("Buffer too large") | |||
There was a problem hiding this comment.
nit: add some detail for better debugging?
There was a problem hiding this comment.
what do you think would be useful here? it's tagged with namespace and schedule id already, which should be enough for someone to find the workload that's causing this. I could add the start time, overlap policy, manual?
There was a problem hiding this comment.
could add schedule spec, start time, buffer size etc.
There was a problem hiding this comment.
I'm a little reluctant to just dump the spec in here since it might be large, and this log might end up repeated a lot. Maybe we should log it once per schedule start so it can be referenced?
Buffer size will always just be the limit here so that's not that useful.
I can log the other args I guess.
service/worker/scheduler/workflow.go
Outdated
| @@ -673,6 +682,11 @@ func (s *scheduler) resolveOverlapPolicy(overlapPolicy enumspb.ScheduleOverlapPo | |||
|
|
|||
| func (s *scheduler) addStart(nominalTime, actualTime time.Time, overlapPolicy enumspb.ScheduleOverlapPolicy, manual bool) { | |||
| s.logger.Debug("addStart", "nominal", nominalTime, "actual", actualTime, "overlapPolicy", overlapPolicy, "manual", manual) | |||
| if s.tweakables.MaxBufferSize > 0 && len(s.State.BufferedStarts) >= s.tweakables.MaxBufferSize { | |||
| s.logger.Error("Buffer too large") | |||
There was a problem hiding this comment.
could add schedule spec, start time, buffer size etc.
What changed?
Why?
Observability
How did you test it?
existing tests, mostly just adding metrics
Potential risks
Is hotfix candidate?