Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is 0 and its equivalent assertions don't work as expected on metrics #1426

Closed
alexggh opened this issue Oct 18, 2023 · 0 comments · Fixed by #1427
Closed

is 0 and its equivalent assertions don't work as expected on metrics #1426

alexggh opened this issue Oct 18, 2023 · 0 comments · Fixed by #1427

Comments

@alexggh
Copy link

alexggh commented Oct 18, 2023

Issue Description

While writing some new zombienet test I discovered that is 0 and its equivalents eguals 0, == 0, = 0 assertion passes even if the metric is non zero, see bellow logs:

 zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +3m
  zombie::network-node returning: 0 +0ms
  zombie::network-node using comparator isAbove for 0, 1 +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:13:43 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +4m
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +53ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator isAbove for 106, 1 +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:13:43 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total is greater than 1 within 5 seconds … │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +2ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:13:44 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +43ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 2  time:  Wed Oct 18 2023 13:13:45 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +46ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 3  time:  Wed Oct 18 2023 13:13:46 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +50ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 4  time:  Wed Oct 18 2023 13:13:47 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +53ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:13:48 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total equals 0 within 5 seconds (5001ms)   │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +814ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node fetching metrics - q: 5  time:  Wed Oct 18 2023 13:13:48 GMT+0300 (Eastern European Summer Time) +187ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +20ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:13:49 GMT+0300 (Eastern European Summer Time) +793ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +813ms
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +44ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 2  time:  Wed Oct 18 2023 13:13:50 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +52ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 3  time:  Wed Oct 18 2023 13:13:51 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +52ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 4  time:  Wed Oct 18 2023 13:13:52 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +36ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:13:53 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total is equal to 0 within 5 seconds (500… │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +821ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node fetching metrics - q: 5  time:  Wed Oct 18 2023 13:13:53 GMT+0300 (Eastern European Summer Time) +180ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +22ms
  zombie::network-node returning: 106 +0ms
  zombie::network-node using comparator equal for 106, 0 +0ms
  zombie::network-node current value: 106 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:13:54 GMT+0300 (Eastern European Summer Time) +798ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +820ms
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +22ms
  zombie::network-node returning: 110 +0ms
  zombie::network-node using comparator equal for 110, 0 +0ms
  zombie::network-node current value: 110 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 2  time:  Wed Oct 18 2023 13:13:55 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +36ms
  zombie::network-node returning: 120 +0ms
  zombie::network-node using comparator equal for 120, 0 +0ms
  zombie::network-node current value: 120 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 3  time:  Wed Oct 18 2023 13:13:56 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +41ms
  zombie::network-node returning: 120 +0ms
  zombie::network-node using comparator equal for 120, 0 +0ms
  zombie::network-node current value: 120 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 4  time:  Wed Oct 18 2023 13:13:57 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +22ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node current value: 125 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:13:58 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total is 0 within 5 seconds (5000ms)       │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +882ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node fetching metrics - q: 5  time:  Wed Oct 18 2023 13:13:58 GMT+0300 (Eastern European Summer Time) +118ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +42ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node current value: 125 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:13:59 GMT+0300 (Eastern European Summer Time) +840ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +883ms
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +47ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node current value: 125 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 2  time:  Wed Oct 18 2023 13:14:00 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +48ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node current value: 125 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 3  time:  Wed Oct 18 2023 13:14:01 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +32ms
  zombie::network-node returning: 125 +0ms
  zombie::network-node using comparator equal for 125, 0 +0ms
  zombie::network-node current value: 125 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 4  time:  Wed Oct 18 2023 13:14:02 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +39ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:14:03 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total == 0 within 5 seconds (5000ms)       │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +837ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node fetching metrics - q: 5  time:  Wed Oct 18 2023 13:14:03 GMT+0300 (Eastern European Summer Time) +163ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +36ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 1  time:  Wed Oct 18 2023 13:14:04 GMT+0300 (Eastern European Summer Time) +801ms
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +836ms
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +38ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 2  time:  Wed Oct 18 2023 13:14:05 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +39ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 3  time:  Wed Oct 18 2023 13:14:06 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +44ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
  zombie::network-node fetching metrics - q: 4  time:  Wed Oct 18 2023 13:14:07 GMT+0300 (Eastern European Summer Time) +1s
  zombie::metrics fetching: http://127.0.0.1:39489/metrics +1s
  zombie::network-node returning for: polkadot_parachain_approvals_no_shows_total from ns: _raw +52ms
  zombie::network-node returning: 127 +0ms
  zombie::network-node using comparator equal for 127, 0 +0ms
  zombie::network-node current value: 127 for metric polkadot_parachain_approvals_no_shows_total, keep trying... +0ms
┌──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 10/18/2023, 1:14:08 PM       │ ✅ alice: reports polkadot_parachain_approvals_no_shows_total = 0 within 5 seconds (5001ms)        │

From @pepoviola quote:

well, to explain the logic we add to cover some corner cases:

we try to assert that a value is equal to 0 between x amount of seconds, IFF we have a timeout and the comparator is equal and the desired value is 0 we make the assertion pass
this was mostly for cases where the value wasn't present in prometheus (because the value was never set)
but for your case that the value is present in prometheus this is a bug
and we should track that the value is present and don't pass the assertion

Steps to reproduce the issue

Steps to reproduce the issue

  1. Try asserting is 0 for a known non-zero metric, e.g:alice: reports polkadot_parachain_approvals_no_shows_total is 0

Describe the results you received

Assertion passed.

Describe the results you expected

Assertion fails

Zombienet version

1.3.69

Provider

Native

Provider version

N/A

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Screenshots

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant