fix: use better value for threshold value in alert description

[srikanthccv]

Summary

The alert sends a description, the threshold is set to X, and the observed metric value is Y

If the condition match type is

• InTotal: the observed metric value computed as sum(values...) ✅
• OnAverage: the observed metric value computed as avg(values...) ✅
• AtleastOnce: the observed metric value computed is whichever is the first value that meets the condition
  Example:
  • ValueIsAbove: any value (usually the first ordered by ts desc) in series > threshold
  • ValueIsBelow: any value (usually the first ordered by ts desc) in series < threshold
  • ValueIsEq: any value (usually the first ordered by ts desc) in series = threshold
  • ValueIsNotEq: any value (usually the first ordered by ts desc) in series != threshold

However, the current logic for AllTheTimes uses the targetVal which doesn't give a good idea of the observed metric value

signoz/pkg/query-service/rules/threshold_rule.go

Line 1200 in 709c286

alertSmpl = Sample{Point: Point{V: r.targetVal()}, Metric: lblsNormalized, MetricOrig: lbls}

This change updates the value to use in the description.

When:

• ValueIsAbove: since we can't show multiple values in the description, we use the min among the series values
• ValueIsBelow: since we can't show multiple values in the description, we use the max among the series values
• ValueIsEq: nothing needs to change
• ValueIsNotEq: any non-inf or nan value from the series

[github-actions]

Build Error! No Linked Issue found. Please link an issue or mention it in the body using #<issue_id>