feat: capture env variables into sigleton and reusable constants #3047

natanasow · 2024-10-01T12:09:03Z

Description:

Currently, we are dynamically getting the environment variables and parsing/casting them every time. This is inefficient and also, more importantly, allows env configurations to be dynamically changed, which can modify the behavior of the relay based on the current values of the env vars, which can be a potential exploit.

Also, when updating CI we noticed with the configuration updates, that if an environment variable such as OPERATOR_ID was defined with not value, it would override the values in the .env file. This caused tests to fail when the relay tried to submit a transaction. Ideally, the relay to fail fast, during bootstrap if required environment variables are not set.

Solution:

Consolidate environment variable loading to the start of the relay. Fail to start if any required environment variables are not set. Better to fail fast.

All environment variables should be handled in one class that can also validate them, list their usage in a --help option, and exit appropriately if a validation fails.

We should also replace environment variables with a yml settings file.

Notes for reviews:

created a new package that holds the config-service singleton
the config-service package is added as a dependency to relay, server, and ws-server
in the perfect world, the config-service must be immutable and should not provide an interface for overriding or extending once-loaded environment variables but there are many tests where we dynamically override envs in order to test specific cases so we should ConfigServiceTestHelper as well
the main review should be applied to the config-service package
almost all of the changes are just replaced values from process.env.<VAR_NAME> to ConfigService.get(<VAR_NAME>)

Needed validations:

Make sure that CHAIN_ID, HEDERA_NETWORK, MIRROR_NODE_URL, OPERATOR_ID_MAIN, OPERATOR_KEY_MAIN, SERVER_PORT are non-empty and in the expected format
Make sure that if OPERATOR_KEY_FORMAT is not specified, the provided OPERATOR_KEY_MAIN is in the DER format
Make sure that HBAR_RATE_LIMIT_TINYBAR is more than HBAR_RATE_LIMIT_BASIC, HBAR_RATE_LIMIT_EXTENDED, HBAR_RATE_LIMIT_PRIVILEGED

Implementation Steps

Consolidate all environment variables into one service class
Add validations and exit on fail of validation, to the environment service class
Add help/usage message and option
There are several solid proposals by @AlfredoG87 that we should take into account as well, described here.
Enhance to use yml instead of dotenv.

cc: @ebadiere

Related issue(s):

Fixes #3023 #2686

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: nikolay <n.atanasow94@gmail.com>

# Conflicts: # packages/relay/src/lib/services/hbarLimitService/index.ts # packages/relay/tests/lib/services/hbarLimitService/hbarLimitService.spec.ts

Signed-off-by: nikolay <n.atanasow94@gmail.com>

# Conflicts: # packages/relay/src/lib/constants.ts # packages/relay/src/lib/eth.ts # packages/relay/tests/lib/eth/eth_call.spec.ts # packages/relay/tests/lib/ethGetBlockBy.spec.ts # packages/relay/tests/lib/repositories/hbarLimiter/ethAddressHbarSpendingPlanRepository.spec.ts # packages/relay/tests/lib/repositories/hbarLimiter/hbarSpendingPlanRepository.spec.ts # packages/relay/tests/lib/services/cacheService/cacheService.spec.ts

Signed-off-by: nikolay <n.atanasow94@gmail.com>

# Conflicts: # packages/relay/src/lib/services/hbarLimitService/index.ts

# Conflicts: # packages/relay/src/lib/clients/sdkClient.ts # packages/relay/tests/lib/hbarLimiter.spec.ts # packages/relay/tests/lib/sdkClient.spec.ts # packages/server/tests/acceptance/hbarLimiter.spec.ts

Signed-off-by: nikolay <n.atanasow94@gmail.com>

# Conflicts: # packages/relay/src/lib/services/hapiService/hapiService.ts # packages/server/tests/acceptance/hbarLimiter.spec.ts # packages/server/tests/acceptance/index.spec.ts # packages/server/tests/acceptance/rpc_batch3.spec.ts # packages/ws-server/tests/acceptance/index.spec.ts

Signed-off-by: nikolay <n.atanasow94@gmail.com>

AlfredoG87

LGTM

Signed-off-by: nikolay <n.atanasow94@gmail.com>

sonarcloud · 2024-10-10T10:02:12Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.6% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-10-10T10:07:21Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled should fail with empty TracerConfig containing invalid properties. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 28,445.1 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 2.10 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 32.77 KB
Total Available Size: increased with 371.38 KB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 64.00 bytes
Used Heap Size: decreased with 822.68 KB
Heap Size Limit: no changes
Malloced Memory: increased with 65.78 KB
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 2.10 MB
- Space Used Size: decreased with 675.31 KB
- Space Available Size: increased with 1.71 MB
- Physical Space Size: increased with 32.77 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

github-actions · 2024-10-10T10:07:45Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled throws an error if block hash is larger than 32bytes. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 27,020.3 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 1.05 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 274.43 KB
Total Available Size: decreased with 1.03 MB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 32.00 bytes
Used Heap Size: decreased with 246.06 KB
Heap Size Limit: no changes
Malloced Memory: no changes
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 1.05 MB
- Space Used Size: decreased with 63.90 KB
- Space Available Size: increased with 63.90 KB
- Physical Space Size: increased with 274.43 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

github-actions · 2024-10-10T10:07:47Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled throws an error for input: {"onlyTopCall":"invalid"}. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 26,883.1 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 2.10 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 32.77 KB
Total Available Size: decreased with 729.38 KB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 32.00 bytes
Used Heap Size: decreased with 95.80 KB
Heap Size Limit: no changes
Malloced Memory: no changes
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 2.10 MB
- Space Used Size: decreased with 74.46 KB
- Space Available Size: increased with 1.11 MB
- Physical Space Size: increased with 32.77 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

github-actions · 2024-10-10T10:08:11Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled does not throw an error for input: {}. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 27,113.8 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 1.05 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 274.43 KB
Total Available Size: decreased with 862.94 KB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 32.00 bytes
Used Heap Size: decreased with 186.54 KB
Heap Size Limit: no changes
Malloced Memory: no changes
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 1.05 MB
- Space Used Size: decreased with 54.84 KB
- Space Available Size: increased with 54.84 KB
- Physical Space Size: increased with 274.43 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

github-actions · 2024-10-10T10:08:14Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled should create a ParseError with correct message and code. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 25,663.8 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 2.10 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 32.77 KB
Total Available Size: increased with 228.49 KB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 32.00 bytes
Used Heap Size: decreased with 247.48 KB
Heap Size Limit: no changes
Malloced Memory: no changes
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 2.10 MB
- Space Used Size: decreased with 78.63 KB
- Space Available Size: increased with 1.11 MB
- Physical Space Size: increased with 32.77 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

github-actions · 2024-10-10T10:08:39Z

🚨 Memory Leak Detected 🚨

A potential memory leak has been detected in the test titled should return a valid JSON-RPC response with error. This may impact the application's performance and stability.

Details

📊 Memory Leak Detection Report 📊

GC Type: MarkSweepCompact
Cost: 34,038.2 ms

Heap Statistics (before vs after executing the test):

Total Heap Size: increased with 1.05 MB
Total Heap Size Executable: no changes
Total Physical Size: increased with 274.43 KB
Total Available Size: decreased with 1.05 MB
Total Global Handles Size: no changes
Used Global Handles Size: decreased with 32.00 bytes
Used Heap Size: decreased with 167.85 KB
Heap Size Limit: no changes
Malloced Memory: no changes
External Memory: no changes
Peak Malloced Memory: no changes

Heap Space Statistics (before vs after executing the test):

New Space:
- Space Size: increased with 1.05 MB
- Space Used Size: decreased with 65.73 KB
- Space Available Size: increased with 65.73 KB
- Physical Space Size: increased with 274.43 KB

Recommendations

Please investigate the memory allocations in this test, focusing on objects that are not being properly deallocated.

codecov · 2024-10-10T10:10:20Z

Codecov Report

Attention: Patch coverage is 81.72589% with 36 lines in your changes missing coverage. Please review.

Project coverage is 85.13%. Comparing base (4a69be9) to head (99be3d0).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
.../relay/src/lib/services/hapiService/hapiService.ts	23.07%	4 Missing and 6 partials ⚠️
packages/relay/src/lib/constants.ts	54.54%	0 Missing and 5 partials ⚠️
packages/relay/src/lib/clients/mirrorNodeClient.ts	81.81%	0 Missing and 4 partials ⚠️
packages/relay/src/lib/eth.ts	42.85%	0 Missing and 4 partials ⚠️
...s/server/src/koaJsonRpc/lib/methodConfiguration.ts	62.50%	0 Missing and 3 partials ⚠️
packages/relay/src/lib/clients/sdkClient.ts	71.42%	1 Missing and 1 partial ⚠️
packages/relay/src/lib/poller.ts	33.33%	0 Missing and 2 partials ⚠️
packages/relay/src/lib/relay.ts	66.66%	0 Missing and 2 partials ⚠️
...kages/config-service/src/services/loggerService.ts	83.33%	0 Missing and 1 partial ⚠️
...kages/relay/src/lib/clients/cache/localLRUCache.ts	66.66%	0 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3047      +/-   ##
==========================================
+ Coverage   84.88%   85.13%   +0.24%     
==========================================
  Files          43       63      +20     
  Lines        3216     4015     +799     
  Branches      649      803     +154     
==========================================
+ Hits         2730     3418     +688     
- Misses        283      358      +75     
- Partials      203      239      +36

Flag	Coverage Δ
config-service	`98.27% <98.27%> (?)`
relay	`85.05% <71.55%> (+0.16%)`	⬆️
server	`83.48% <84.00%> (?)`
ws-server	`97.91% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ckages/config-service/src/services/globalConfig.ts	`100.00% <100.00%> (ø)`
packages/config-service/src/services/index.ts	`100.00% <100.00%> (ø)`
...s/config-service/src/services/validationService.ts	`100.00% <100.00%> (ø)`
packages/relay/src/lib/hbarlimiter/index.ts	`100.00% <100.00%> (ø)`
packages/relay/src/lib/net.ts	`100.00% <100.00%> (+12.50%)`	⬆️
...elay/src/lib/services/cacheService/cacheService.ts	`92.53% <100.00%> (+0.05%)`	⬆️
...kages/relay/src/lib/services/debugService/index.ts	`76.81% <100.00%> (+0.34%)`	⬆️
.../lib/services/ethService/ethCommonService/index.ts	`89.23% <100.00%> (+0.08%)`	⬆️
.../lib/services/ethService/ethFilterService/index.ts	`79.76% <100.00%> (+0.24%)`	⬆️
...ay/src/lib/services/metricService/metricService.ts	`92.68% <100.00%> (+0.18%)`	⬆️
... and 22 more

... and 7 files with indirect coverage changes

natanasow added 30 commits September 10, 2024 14:21

chore: refactor packages

78629c8

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix ts error

620f19f

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix import

3927017

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: edit acceptance tests

b6160fa

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: edit tests

191efd3

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix unit tests

e47fdc7

Signed-off-by: nikolay <n.atanasow94@gmail.com>

Merge branch 'main' into 2686-capture-env-variables-into-constants

7bc5a10

chore: fix unit tests

0127061

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix namespaces

442bf62

Signed-off-by: nikolay <n.atanasow94@gmail.com>

Merge branch 'main' into 2686-capture-env-variables-into-constants

d75063a

# Conflicts: # packages/relay/src/lib/services/hbarLimitService/index.ts # packages/relay/tests/lib/services/hbarLimitService/hbarLimitService.spec.ts

chore: fix ws server tests

e92423a

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix server namespaces

35de633

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix typo

6dc109b

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix hbar limiter tests

fbf978b

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: edit ws server imports

eae9429

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix hbar limiter spec

3b2c5e8

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: add env-provider package

563c7f2

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: resolve conflicts

4297965

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix tests

cf871df

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: remove unused deps

8afa33c

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix comment

2cd2671

Signed-off-by: nikolay <n.atanasow94@gmail.com>

Merge branch 'main' into 2686-capture-env-variables-into-constants

8ba4974

# Conflicts: # packages/relay/src/lib/services/hbarLimitService/index.ts

Merge branch 'main' into 2686-capture-env-variables-into-constants

5eab792

# Conflicts: # packages/relay/src/lib/clients/sdkClient.ts # packages/relay/tests/lib/hbarLimiter.spec.ts # packages/relay/tests/lib/sdkClient.spec.ts # packages/server/tests/acceptance/hbarLimiter.spec.ts

chore: resolve conflicts

ff24221

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: resolve commets

db2bc96

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: fix type

be3a4b5

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: remove unused ts-ignore

483c6e3

Signed-off-by: nikolay <n.atanasow94@gmail.com>

chore: resolve conflicts

788b105

Signed-off-by: nikolay <n.atanasow94@gmail.com>

hashgraph deleted a comment from github-actions bot Oct 8, 2024

AlfredoG87 previously approved these changes Oct 8, 2024

View reviewed changes

chore: add license headers

99be3d0

Signed-off-by: nikolay <n.atanasow94@gmail.com>

natanasow dismissed AlfredoG87’s stale review via 99be3d0 October 10, 2024 10:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: capture env variables into sigleton and reusable constants #3047

feat: capture env variables into sigleton and reusable constants #3047

natanasow commented Oct 1, 2024 •

edited

Loading

AlfredoG87 left a comment

sonarcloud bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 10, 2024

codecov bot commented Oct 10, 2024

feat: capture env variables into sigleton and reusable constants #3047

Are you sure you want to change the base?

feat: capture env variables into sigleton and reusable constants #3047

Conversation

natanasow commented Oct 1, 2024 • edited Loading

Implementation Steps

AlfredoG87 left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Oct 10, 2024

Quality Gate passed

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

github-actions bot commented Oct 10, 2024

🚨 Memory Leak Detected 🚨

Details

Recommendations

codecov bot commented Oct 10, 2024

Codecov Report

natanasow commented Oct 1, 2024 •

edited

Loading