enhance: [cherry-pick] Add monitoring metrics for task execution time in datacoord #35141
Conversation
mergify
bot
added
dco-passed
internal/datacoord/task_scheduler.go
Outdated
| case indexpb.JobState_JobStateNone: | ||
| return | ||
| case indexpb.JobState_JobStateInit: | ||
| queueTime := time.Since(task.GetQueueTime()) |
There was a problem hiding this comment.
Please log warn if the pending or executing time of a task with a given taskID exceeds 6 hours ?
There was a problem hiding this comment.
I think 6 hours is too long, as it may delay the timely detection of issues. However, I made this duration configurable so it can be adjusted as needed. default duration is 5 minutes.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.4 #35141 +/- ##
==========================================
+ Coverage 72.12% 72.16% +0.03%
==========================================
Files 1052 1052
Lines 136846 136958 +112
==========================================
+ Hits 98707 98832 +125
+ Misses 33925 33912 -13
Partials 4214 4214
|
xiaocai2333
force-pushed
the
support_queue_time-4
branch
from
7394ef7
to
b67bf38
Compare
| log.Warn("task running time is too long", zap.Int64("taskID", taskID), | ||
| zap.Int64("running time(ms)", runningTime.Milliseconds())) | ||
| } | ||
| metrics.DataCoordTaskExecuteLatency. |
There was a problem hiding this comment.
why we report this every time?
I believe we only need to report this once when index excution done?
There was a problem hiding this comment.
It will only be report once here. This is the task queue, and when a task is finished and the meta is successfully recorded, the task will be cleared from the queue.
internal/datacoord/task_scheduler.go
Outdated
|
|
||
| task := s.getTask(taskID) | ||
| if task == nil { | ||
| s.taskLock.Unlock(taskID) |
There was a problem hiding this comment.
no need to unblock it?
There was a problem hiding this comment.
yeah, it's wrong, will remove it.
xiaocai2333
force-pushed
the
support_queue_time-4
branch
from
f691dd3
to
d6539e4
Compare
xiaocai2333
force-pushed
the
support_queue_time-4
branch
from
d6539e4
to
56a1301
Compare
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
xiaocai2333
force-pushed
the
support_queue_time-4
branch
from
56a1301
to
5154b9b
Compare
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: czs007, xiaocai2333 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
issue: #35138
master pr: #35139