query-tee: add equivalent errors for string expression for range quer…

…ies (#9366) (#9444)

* query-tee: add equivalent errors for string expression for range queries

* Add changelog entry

(cherry picked from commit 47aae6a)

Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>

CHANGELOG.md

@@ -215,7 +215,7 @@
 * [ENHANCEMENT] Don't consider responses to be different during response comparison if both backends' responses contain different series, but all samples are within the recent sample window. #8749 #8894
 * [ENHANCEMENT] When the expected and actual response for a matrix series is different, the full set of samples for that series from both backends will now be logged. #8947
 * [ENHANCEMENT] Wait up to `-server.graceful-shutdown-timeout` for inflight requests to finish when shutting down, rather than immediately terminating inflight requests on shutdown. #8985
-* [ENHANCEMENT] Optionally consider equivalent error messages the same when comparing responses. Enabled by default, disable with `-proxy.require-exact-error-match=true`. #9143 #9350
+* [ENHANCEMENT] Optionally consider equivalent error messages the same when comparing responses. Enabled by default, disable with `-proxy.require-exact-error-match=true`. #9143 #9350 #9366
 * [BUGFIX] Ensure any errors encountered while forwarding a request to a backend (eg. DNS resolution failures) are logged. #8419
 * [BUGFIX] The comparison of the results should not fail when either side contains extra samples from within SkipRecentSamples duration. #8920
 * [BUGFIX] When `-proxy.compare-skip-recent-samples` is enabled, compare sample timestamps with the time the query requests were made, rather than the time at which the comparison is occurring. #9416

tools/querytee/response_comparator.go

@@ -120,12 +120,19 @@ func (s *SamplesComparator) Compare(expectedResponse, actualResponse []byte, que
 
 var errorEquivalenceClasses = [][]*regexp.Regexp{
 	{
-		// Invalid expression type for range query: MQE and Prometheus' engine return different error messages.
+		// Range vector expression for range query: MQE and Prometheus' engine return different error messages.
 		// Prometheus' engine:
 		regexp.MustCompile(`invalid parameter "query": invalid expression type "range vector" for range query, must be Scalar or instant Vector`),
 		// MQE:
 		regexp.MustCompile(`invalid parameter "query": query expression produces a range vector, but expression for range queries must produce an instant vector or scalar`),
 	},
+	{
+		// String expression for range query: MQE and Prometheus' engine return different error messages.
+		// Prometheus' engine:
+		regexp.MustCompile(`invalid parameter "query": invalid expression type "string" for range query, must be Scalar or instant Vector`),
+		// MQE:
+		regexp.MustCompile(`invalid parameter "query": query expression produces a string, but expression for range queries must produce an instant vector or scalar`),
+	},
 	{
 		// Binary operation conflict on right (one-to-one) / many (one-to-many/many-to-one) side: MQE and Prometheus' engine return different error messages, and there's no guarantee they'll pick the same series as examples.
 		// Even comparing Prometheus' engine to another instance of Prometheus' engine can produce different results: the series selected as examples are not deterministic.