Add IP metrics to REST #2174

ijungmann · 2021-06-24T17:47:40Z

Detailed description:
Add a metric to track which IPs hit which endpoints.

The intention is to use this metric for environments where we have whitelisted IPs and already know who is accessing the API, it is not meant for public environments.

Which issue(s) this PR fixes:
Fixes #2130

Special notes for your reviewer:

Checklist

Documentation added
Tests updated

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

codecov · 2021-06-24T17:50:01Z

Codecov Report

❗ No coverage uploaded for pull request base (main@305413f). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2174   +/-   ##
=======================================
  Coverage        ?   81.82%           
  Complexity      ?     2130           
=======================================
  Files           ?      414           
  Lines           ?    11131           
  Branches        ?      945           
=======================================
  Hits            ?     9108           
  Misses          ?     1714           
  Partials        ?      309

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 305413f...687f89e. Read the comment docs.

hedera-mirror-rest/accounts.js

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

hedera-mirror-rest/middleware/metricsHandler.js

ijungmann · 2021-06-25T04:49:22Z

A sample of what the metrics will look like (tested locally).
ip_count_bucket{le="0.1",endpoint="/api/v1/transactions/:id",ip="::1"} 0 ip_count_bucket{le="5",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="15",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="50",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="100",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="500",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="+Inf",endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_sum{endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_count{endpoint="/api/v1/transactions/:id",ip="::1"} 1 ip_count_bucket{le="0.1",endpoint="/api/v1/accounts",ip="::1"} 0 ip_count_bucket{le="5",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="15",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="50",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="100",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="500",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="+Inf",endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_sum{endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_count{endpoint="/api/v1/accounts",ip="::1"} 1 ip_count_bucket{le="0.1",endpoint="/api/v1/accounts/:id",ip="::1"} 0 ip_count_bucket{le="5",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_bucket{le="15",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_bucket{le="50",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_bucket{le="100",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_bucket{le="500",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_bucket{le="+Inf",endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_sum{endpoint="/api/v1/accounts/:id",ip="::1"} 1 ip_count_count{endpoint="/api/v1/accounts/:id",ip="::1"} 1

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Nana-EC

Clarification requests

hedera-mirror-rest/server.js

hedera-mirror-rest/middleware/metricsHandler.js

hedera-mirror-rest/transactions.js

hedera-mirror-rest/middleware/metricsHandler.js

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann · 2021-06-25T21:07:53Z

After changes, this is the look:
hedera_mirror_rest_request_count{endpoint="/api/v1/transactions",ip="eff8e7ca506627fe15dda5e0e512fcaad70b6d520f37cc76597fdb4f2d83a1a3"} 2 hedera_mirror_rest_request_count{endpoint="/api/v1/transactions/:id",ip="eff8e7ca506627fe15dda5e0e512fcaad70b6d520f37cc76597fdb4f2d83a1a3"} 2 hedera_mirror_rest_request_count{endpoint="/api/v1/tokens",ip="eff8e7ca506627fe15dda5e0e512fcaad70b6d520f37cc76597fdb4f2d83a1a3"} 1 hedera_mirror_rest_request_count{endpoint="/api/v1/tokens/:id/balances",ip="eff8e7ca506627fe15dda5e0e512fcaad70b6d520f37cc76597fdb4f2d83a1a3"} 1

hedera-mirror-rest/middleware/metricsHandler.js

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann · 2021-06-28T17:38:17Z

hedera-mirror-rest/package-lock.json

@@ -1,8 +1,11722 @@
 {
  "name": "hedera-mirror-rest",
  "version": "0.37.0-SNAPSHOT",
-  "lockfileVersion": 1,
+  "lockfileVersion": 2,


Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

hedera-mirror-rest/middleware/metricsHandler.js

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann · 2021-06-28T17:55:31Z

hedera-mirror-rest/__tests__/utils.test.js

+describe('Utils ipMask tests', () => {
+  test('Verify ipV4', () => {
+    const maskedIp = utils.ipMask('12.214.31.144');
+    expect(maskedIp).toStrictEqual('12.214.31.0');
+  });
+  test('Verify ipV6', () => {
+    const maskedIp = utils.ipMask('2001:0db8:85a3:a13c:0000:8a2e:0370:7334');
+    expect(maskedIp).toStrictEqual('2001:db8:85a3::');
+  });
+  test('Verify ipV6 short form back', () => {
+    const maskedIp = utils.ipMask('1::');
+    expect(maskedIp).toStrictEqual('1::');
+  });
+  test('Verify ipV6 short form front', () => {
+    const maskedIp = utils.ipMask('::ffff');
+    expect(maskedIp).toStrictEqual('::');
+  });
+  test('Verify ipV6 dual', () => {
+    const maskedIp = utils.ipMask('2001:db8:3333:4444:5555:6666:1.2.3.4');
+    expect(maskedIp).toStrictEqual('2001:db8:3333::0.0.0.0');
+  });


These IPs were taken from examples for the library, some made up, none are sensitive.

steven-sheehy

LGTM

xin-hedera

LGTM

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Nana-EC

LGTM

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

xin-hedera

LGTM

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

sonarcloud · 2021-06-28T22:37:54Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
9 Security Hotspots
28 Code Smells

0.0% Coverage
4.8% Duplication

First attempt at adding ip metrics

4c981b7

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann added this to the Mirror 0.37.0 milestone Jun 24, 2021

ijungmann added enhancement Type: New feature P2 rest Area: REST API labels Jun 24, 2021

steven-sheehy reviewed Jun 24, 2021

View reviewed changes

hedera-mirror-rest/accounts.js Outdated Show resolved Hide resolved

Ian Jungmann added 5 commits June 24, 2021 15:34

Move everything into middleware

ea2ca0a

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Move everything into middleware

9e43123

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Put auth back

9e451a9

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Move things around, document, add config, cleanup

b3cdf12

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Cleanup

00417bd

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann commented Jun 25, 2021

View reviewed changes

hedera-mirror-rest/middleware/metricsHandler.js Outdated Show resolved Hide resolved

ijungmann marked this pull request as ready for review June 25, 2021 04:52

Rename metric

644dec8

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann requested a review from a team June 25, 2021 17:30

Nana-EC assigned ijungmann Jun 25, 2021

Nana-EC requested changes Jun 25, 2021

View reviewed changes

hedera-mirror-rest/server.js Show resolved Hide resolved

steven-sheehy requested changes Jun 25, 2021

View reviewed changes

Ian Jungmann added 2 commits June 25, 2021 16:05

Make metric a counter and hash

efa4e75

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

var to const

1b15324

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann requested review from steven-sheehy and Nana-EC June 25, 2021 21:07

steven-sheehy reviewed Jun 25, 2021

View reviewed changes

hedera-mirror-rest/middleware/metricsHandler.js Outdated Show resolved Hide resolved

xin-hedera requested changes Jun 25, 2021

View reviewed changes

hedera-mirror-rest/middleware/metricsHandler.js Outdated Show resolved Hide resolved

Switch to ip masking

93ec060

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann requested review from steven-sheehy and xin-hedera June 28, 2021 17:33

ijungmann commented Jun 28, 2021

View reviewed changes

Ian Jungmann added 2 commits June 28, 2021 12:45

Fix package-lock

04e774b

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

fix package-lock

50a7cdf

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

steven-sheehy reviewed Jun 28, 2021

View reviewed changes

hedera-mirror-rest/middleware/metricsHandler.js Outdated Show resolved Hide resolved

Remove dead code

55296e3

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann commented Jun 28, 2021

View reviewed changes

steven-sheehy previously approved these changes Jun 28, 2021

View reviewed changes

xin-hedera previously approved these changes Jun 28, 2021

View reviewed changes

Merge with main

ab06b0d

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann dismissed stale reviews from xin-hedera and steven-sheehy via ab06b0d June 28, 2021 19:26

Nana-EC previously approved these changes Jun 28, 2021

View reviewed changes

Fix swagger stats

f035ea3

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann dismissed Nana-EC’s stale review via f035ea3 June 28, 2021 20:43

Move swagger config stuff to other PR

e17d282

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann requested review from steven-sheehy, Nana-EC and xin-hedera June 28, 2021 20:53

xin-hedera previously approved these changes Jun 28, 2021

View reviewed changes

steven-sheehy previously approved these changes Jun 28, 2021

View reviewed changes

Nana-EC previously approved these changes Jun 28, 2021

View reviewed changes

Move new library into correct place in package.json

687f89e

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann dismissed stale reviews from Nana-EC, steven-sheehy, and xin-hedera via 687f89e June 28, 2021 22:33

steven-sheehy approved these changes Jun 28, 2021

View reviewed changes

ijungmann merged commit 0e4e0d0 into main Jun 28, 2021

ijungmann deleted the Add_rest_ip_metrics branch June 28, 2021 22:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IP metrics to REST #2174

Add IP metrics to REST #2174

ijungmann commented Jun 24, 2021 •

edited

Loading

codecov bot commented Jun 24, 2021 •

edited

Loading

ijungmann commented Jun 25, 2021

Nana-EC left a comment

ijungmann commented Jun 25, 2021

ijungmann Jun 28, 2021

ijungmann Jun 28, 2021

steven-sheehy left a comment

xin-hedera left a comment

Nana-EC left a comment

xin-hedera left a comment

sonarcloud bot commented Jun 28, 2021

Add IP metrics to REST #2174

Add IP metrics to REST #2174

Conversation

ijungmann commented Jun 24, 2021 • edited Loading

codecov bot commented Jun 24, 2021 • edited Loading

Codecov Report

ijungmann commented Jun 25, 2021

Nana-EC left a comment

Choose a reason for hiding this comment

ijungmann commented Jun 25, 2021

ijungmann Jun 28, 2021

Choose a reason for hiding this comment

ijungmann Jun 28, 2021

Choose a reason for hiding this comment

steven-sheehy left a comment

Choose a reason for hiding this comment

xin-hedera left a comment

Choose a reason for hiding this comment

Nana-EC left a comment

Choose a reason for hiding this comment

xin-hedera left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Jun 28, 2021

ijungmann commented Jun 24, 2021 •

edited

Loading

codecov bot commented Jun 24, 2021 •

edited

Loading