Druid automated quickstart #13365

findingrish · 2022-11-15T05:58:19Z

Description

Setting up a new Druid cluster could involve tuning several runtime properties required by each service. Each process must also be given the appropriate jvm arguments so that the system can perform optimally.

Druid already contains several single-server profiles such as nano-quickstart, micro-quickstart, large-quickstart, etc. But it is still difficult to have a setup which does not match exactly with any of these profiles. A lot of the properties in these profiles can also be commoned out and simplified.

This PR adds a new script start-druid in order to:

simplify starting any combination of Druid services on a single server
reduce the number of inputs required to start Druid
deprecate the existing profiles and eventually remove them

Usage

Command:

./start-druid [--memory=totalAvailableMemory] [--services=listOfServices] [--config=rootConfigDir]

Examples:

Run all services with total system memory and default runtime properties
./start-druid
Run one instance each of specific services with given total memory
./start-druid --services=broker,router,historical --memory=10g
Run services with overridden properties
./start-druid --services=broker,router,middleManager,historical --config=rootConfigDir
Directory _common must be present inside rootConfigDir and must contain files common.runtime.properties and common.jvm.config with mandatory properties and any other property that you wish to override. Please refer to conf/druid/auto/_common for an example.

Override runtime properties of a service: Specify a runtime.properties for that service. For the example command above, create a file rootConfigDir/broker/runtime.properties. This file should contain the properties you wish to override.
Override jvm arguments for a service: To override the jvm args for any service, you must override the jvm args for all the services. This is to prevent miscalculations in allocated memory. For the example command above, create directories broker, router, middleManager, historical inside the rootConfigDir.
Add a file jvm.config specifying the following values in each of the service directories.
- Xmx***
- Xms***
- XX:MaxDirectMemorySize*** (required only for broker and historical)
For middleManager, alongwith jvm args, you must override druid.indexer.runner.javaOptsArray in rootConfigDir/middleManager/runtime.properties

Design

Flow

Run bash script start-druid with one of the possible invocations
start-druid invokes python script start-druid-main.py which computes the memory required for each service

start-druid-main.py invokes bin/supervise with the commands for each service to start.

example broker command:

broker bin/run-druid broker ~/conf/druid/auto '-Xms2392m -Xmx2392m -XX:MaxDirectMemorySize=1595m'

example middle-manager command:

!p90 middleManager bin/run-druid middleManager /apache-druid-{{DRUIDVERSION}}/conf/druid/auto '-Xms346m -Xmx346m' '-Ddruid.worker.capacity=5 -Ddruid.indexer.runner.javaOptsArray=["-server","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","-Xms1g","-Xmx1g","-XX:MaxDirectMemorySize=1g"]'

bin/supervise starts the services, one at a time, by invoking run-druid (same as existing flow while using ./bin/start-micro-quickstart etc.)

Parameters

start-druid accepts these optional parameters:

parameter	description	default value
`memory`	total allocated memory for all Druid services and tasks	80% of total system memory
`config`	root config directory, where service-specific `runtime.properties` and `jvm.config` can be given	`examples/conf/druid/auto`
`services`	list of services to start	all services (including zookeeper)
`zk`	whether to start zookeeper	false
`verbose`	print all messages and computed parameter values	false
`compute`	do not start Druid, only print computed parameter values	false

Validations

memory should be in valid format (integer followed by m or g)
memory should satisfy the total minimum requirements
<rootConfDir>/<service>/jvm.config should either be present for all the services being started or none
<rootConfDir>/middleManager/runtime.properties must have druid.indexer.runner.javaOptsArray parameter, if middleManager is to be started with overridden memory

Memory distribution

service	total memory ratio (heap + direct)	minimum total memory (MB)
middleManager	1	64
router	2	128
coordinator-overlord	30	256
task	30	1024
broker	46	900
historical	80	900

Running existing profiles

nano:  4g,  2 cpu 
micro:  16g, 4 cpu
small: 64g, 8 cpu
medium: 128g, 16 cpu
large: 256g, 32 cpu
xlarge: 512g, 64 cpu

For example, starting small in verbose mode:

> ./start-druid -m=64g --verbose
...
total memory is 65536m
memory used for services & tasks 32768m

Memory distribution for services:
broker, memory_config: -Xms4785m -Xmx4785m -XX:MaxDirectMemorySize=3190m
router, memory_config: -Xms346m -Xmx346m
coordinator-overlord, memory_config: -Xms5201m -Xmx5201m
historical, memory_config: -Xms5548m -Xmx5548m -XX:MaxDirectMemorySize=8322m
middleManager, memory_config: -Xms173m -Xmx173m
middleManager-task, memory_config: ['-Ddruid.worker.capacity=2', '-Ddruid.indexer.runner.javaOptsArray=["-server","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager","-Xms1g","-Xmx1g","-XX:MaxDirectMemorySize=1g"]']

Limitations

can only launch a single instance of each service
does not allow overriding the jvm arguments for just one service, either all must be overridden or none

Major changes

Added start-druid, bash script with functionality as described above
Added start-druid-main, python script with functionality as described above
Added conf/druid/auto directory
Updated supervise, to accept service startup commands as arguments, refactored the method which reads the supervise config file
Update run-druid script to accept optional parameters for jvm argument and middle manager task params

A note on runtime properties

The following runtime properties have not been added in the service-specific runtime.properties files inside conf/druid/auto because their default values computed in the code seem adequate.

property	service	value computed in Druid code
druid.server.http.numThreads	all	`max(10, (numCores * 17) / 16 + 2) + 30`
druid.processing.numThreads	broker, historical	`max(numCores - 1, 1)`
druid.broker.http.numConnections	broker	20
druid.broker.http.maxQueuedBytes	broker	`max(25m, heap_size * 0.02)`
druid.processing.buffer.sizeBytes	broker, historical	`min(1g, directMemory/ (numThreads + numMergeBuffers + 1))`
druid.processing.buffer.numMergeBuffers	broker, historical	`max(2, numThreads / 4)`
druid.cache.sizeInBytes		`~(heap_size / 10)`
druid.worker.capacity	middle-manager	`max(numCores - 1, 1)`
druid.indexer.runner.javaOptsArray	middle-manager	read from `middle-manager/runtime.properties` if specified, otherwise computed by `start-druid`
druid.indexer.fork.property.druid.processing.numMergeBuffers
druid.indexer.fork.property.druid.processing.buffer.sizeBytes
druid.indexer.fork.property.druid.processing.numThreads
druid.router.http.numConnections	router	20
druid.router.http.readTimeout	router	15 mins
druid.router.http.numMaxThreads	router	`max(10, ((numCores * 17) / 16 + 2) + 30)`

Release note

Add a new script start-druid which greatly simplifies starting Druid services on a single server.

Usage:

./start-druid[--memory=totalAvailableMemory] [--services=listOfServices] [--config=rootConfigDir]

This PR has:

lgtm-com · 2022-11-15T06:04:43Z

This pull request introduces 1 alert when merging 8c9dcc2 into 309cae7 - view on LGTM.com

new alerts:

1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-15T06:13:08Z

This pull request introduces 1 alert when merging 342e5d7 into 309cae7 - view on LGTM.com

new alerts:

1 for Unused local variable

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

FrankChen021 · 2022-11-17T03:55:10Z

This is a great work to ease the initial setup work for users especially when they're new to Druid.

This startup script relies on a Python script, I know that writting python scripts is much simpler than writing bash scripts, but my question is that what if they don't have a Python interpreter? Which version of Python do they need to run this script? Python 2 or Python 3? Will they struggle on installing a python interpreter?

findingrish · 2022-11-18T05:18:23Z

This is a great work to ease the initial setup work for users especially when they're new to Druid.

This startup script relies on a Python script, I know that writting python scripts is much simpler than writing bash scripts, but my question is that what if they don't have a Python interpreter? Which version of Python do they need to run this script? Python 2 or Python 3? Will they struggle on installing a python interpreter?

@FrankChen021 thanks for going over the change.

Which version of Python do they need to run this script?

It requires python 3.

Will they struggle on installing a python interpreter?

Installing python is straightforward on most environments, IMO it shouldn't pose any challenge.

…anges in druid-quickstart.py

…tion changes

lgtm-com · 2022-11-18T19:43:46Z

This pull request introduces 1 alert when merging 68a1e66 into 5b625ce - view on LGTM.com

new alerts:

1 for Module is imported with 'import' and 'import from'

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-11-18T20:29:13Z

This pull request introduces 1 alert when merging e570606 into 5b625ce - view on LGTM.com

new alerts:

1 for Module is imported with 'import' and 'import from'

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

gianm

Thanks for the contribution! I haven't reviewed the code, but I do have some comments based on building this branch on my machine and running it:

On my machine (64GB RAM) the script decided to set total memory = 52GB and then allocated that chunk of memory 100% to on-heap and direct memory for the various processes. It doesn't leave any room for page cache or for tasks on MMs. The calculations should be adjusted to take those items into account. (By "take page cache into account", I mean leave some memory free so the OS can use it to cache mmapped segments.) I suggest comparing the calculated numbers here against the single-server configs we have: https://druid.apache.org/docs/latest/operations/single-server.html. For example, 64GB RAM would match our "small" config, where we give the Historical a 4g heap; the script currently generates 12752m.
The way the script accepts command-line arguments is nonstandard; consider using argparse instead and accepting --x=y style arguments. It would help with error messages, too. I tried an invalid totalMemory setting and noticed that the error message had a traceback in it, which makes it look messy. argparse wouldn't do that.
I noticed that when I provide a very low totalMemory (like 40m) the calculations generate negative numbers and then the processes crash. It would be better for the script to print a nice error message.
The script should be less chatty in the normal case. Feel free to provide a --verbose mode, but normally, I don't think it should print anything. The first line the user should see is the one from bin/greet: "Starting Apache Druid."
When you have scripts tail-calling out to other scripts, it's best to use exec (in bash) and execv or one of the other exec variants (in python). This way, each call replaces the original one rather than forming a chain. It is cleaner and uses fewer resources. For the bash script start-druid, refer to post-index-task for an example of how you can use exec. For python refer to https://docs.python.org/3/library/os.html for os.exec* functions.

lgtm-com · 2022-11-21T06:34:05Z

This pull request introduces 1 alert when merging b413d15 into fa3ab27 - view on LGTM.com

new alerts:

1 for Module is imported with 'import' and 'import from'

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

findingrish · 2022-11-21T09:15:12Z

@gianm thanks for the feedback.

On my machine (64GB RAM) the script decided to set total memory = 52GB and then allocated that chunk of memory 100% to on-heap and direct memory for the various processes. It doesn't leave any room for page cache or for tasks on MMs. The calculations should be adjusted to take those items into account. (By "take page cache into account", I mean leave some memory free so the OS can use it to cache mmapped segments.) I suggest comparing the calculated numbers here against the single-server configs we have: https://druid.apache.org/docs/latest/operations/single-server.html. For example, 64GB RAM would match our "small" config, where we give the Historical a 4g heap; the script currently generates 12752m.

After going over https://druid.apache.org/docs/latest/operations/single-server.html
I tried to check the ratio (service memory (heap, direct) + task on MMs) / totalMemory
nano -> (2 + 1)g / 4g
micro -> (3 + 4)g / 16g
small -> (25 + 6)g / 64g
medium -> (44 + 8)g / 128g
large -> (85 + 16)g / 256g
xlarge -> (99 + 32)g / 512g

except for nano and xlarge profiles, the ratio is close to 50%
I am thinking of using 50% as default and allow the user to override this ratio, does that work?

This script also provides user the ability to run any subset of services, which implies one can start standalone services,
does the 50% threshold work for it? or is there a way to estimate the free memory required by individual services (something like broker with 7g of memory (heap + direct) needs x gb of free memory?

kfaraz

Done a partial review. Added some comments.

examples/bin/druid-quickstart.py

examples/bin/run-druid

examples/bin/druid-quickstart.py

supervise script changes to process java opts array use argparse, leave free memory, logging

lgtm-com · 2022-12-08T05:48:47Z

This pull request introduces 3 alerts when merging a4d14f7 into fbf76ad - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Variable defined multiple times

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

lgtm-com · 2022-12-08T06:46:18Z

This pull request introduces 3 alerts when merging 08507b4 into d85fb8c - view on LGTM.com

new alerts:

2 for Unused local variable
1 for Variable defined multiple times

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

vogievetsky · 2022-12-08T21:10:26Z

examples/bin/start-druid

+then
+  exec python2 "$WHEREAMI/start-druid-main.py" "$@"
+else
+  exec python "$WHEREAMI/start-druid-main.py" "$@"


this should do a elif [ -x "$(command -v python)" ] and if that fails (in the else) it should print out a nice warning that python could not be found and is needed

This can be done as a followup also

vogievetsky

Amazing

gianm

LGTM, thank you! Let's remember, as part of the next release cycle, to test this on a bunch of OSes so we can be sure things work as expected.

vogievetsky · 2022-12-09T01:17:43Z

docs/operations/single-server.md

+Druid includes a set of reference configurations and launch scripts for single-machine deployments.
+These configuration bundles are located in `conf/druid/single-server/`.
+
+The `auto` configuration sizes runtime parameters based on available processors and memory. Other configurations include hard-coded runtime parameters for various server sizes. Most users should stick with `auto`. Refer below [Druid auto start](#Druid-auto-start)


I think the anchor needs to be #druid-auto-start not #Druid-auto-start

vogievetsky · 2022-12-09T01:47:02Z

examples/bin/supervise

 );

-usage() unless $opt{'conf'} && $opt{'vardir'};
+usage() unless ((@{$opt{'command'}} || $opt{'conf'}) && $opt{'vardir'});


I don't know enough perl but something in this line is messed up

When I run

./bin/start-micro-quickstart Can't use an undefined value as an ARRAY reference at /Users/vadim/Projects/apache-druid/distribution/target/apache-druid-26.0.0-SNAPSHOT/bin/supervise line 202.

gianm · 2022-12-09T06:53:09Z

examples/bin/supervise

 );

-usage() unless $opt{'conf'} && $opt{'vardir'};
+usage() unless (($opt{'conf'} || @{$opt{'command'}}) && $opt{'vardir'});


Ah, this should be:

usage() unless (($opt{'command'} && @{$opt{'command'}}) || $opt{'conf'}) && $opt{'vardir'}

breaking down ($opt{'command'} && @{$opt{'command'}}):

$opt{'command'} means "there is some value at $opt{'command'}

we expect this to be an arrayref, so, @{$opt{'command'}} means "the arrayref at $opt{'command'} is nonempty"

* Druid automated quickstart * remove conf/druid/single-server/quickstart/_common/historical/jvm.config * Minor changes in python script * Add lower bound memory for some services * Additional runtime properties for services * Update supervise script to accept command arguments, corresponding changes in druid-quickstart.py * File end newline * Limit the ability to start multiple instances of a service, documentation changes * simplify script arguments * restore changes in medium profile * run-druid refactor * compute and pass middle manager runtime properties to run-druid supervise script changes to process java opts array use argparse, leave free memory, logging * Remove extra quotes from mm task javaopts array * Update logic to compute minimum memory * simplify run-druid * remove debug options from run-druid * resolve the config_path provided * comment out service specific runtime properties which are computed in the code * simplify run-druid * clean up docs, naming changes * Throw ValueError exception on illegal state * update docs * rename args, compute_only -> compute, run_zk -> zk * update help documentation * update help documentation * move task memory computation into separate method * Add validation checks * remove print * Add validations * remove start-druid bash script, rename start-druid-main * Include tasks in lower bound memory calculation * Fix test * 256m instead of 256g * caffeine cache uses 5% of heap * ensure min task count is 2, task count is monotonic * update configs and documentation for runtime props in conf/druid/single-server/quickstart * Update docs * Specify memory argument for each profile in single-server.md * Update middleManager runtime.properties * Move quickstart configs to conf/druid/base, add bash launch script, support python2 * Update supervise script * rename base config directory to auto * rename python script, changes to pass repeated args to supervise * remove exmaples/conf/druid/base dir * add docs * restore changes in conf dir * update start-druid-auto * remove hashref for commands in supervise script * start-druid-main java_opts array is comma separated * update entry point script name in python script * Update help docs * documentation changes * docs changes * update docs * add support for running indexer * update supported services list * update help * Update python.md * remove dir * update .spelling * Remove dependency on psutil and pathlib * update docs * Update get_physical_memory method * Update help docs * update docs * update method to get physical memory on python * udpate spelling * update .spelling * minor change * Minor change * memory comptuation for indexer * update start-druid * Update python.md * Update single-server.md * Update python.md * run python3 --version to check if python is installed * Update supervise script * start-druid: echo message if python not found * update anchor text * minor change * Update condition in supervise script * JVM not jvm in docs

Cherry-picked from commit 4ebdfe2

maytasm · 2023-05-06T04:02:22Z

Did we forgot to change the default values in Configuration reference doc page?
For example, the default value of druid.cache.sizeInBytes for caffeine was change from min(1GiB, Runtime.maxMemory / 10) to min(1GiB, Runtime.maxMemory / 20)

findingrish · 2023-07-31T10:50:43Z

Did we forgot to change the default values in Configuration reference doc page? For example, the default value of druid.cache.sizeInBytes for caffeine was change from min(1GiB, Runtime.maxMemory / 10) to min(1GiB, Runtime.maxMemory / 20)

Thanks for pointing it out. I will update it.

Druid automated quickstart

8c9dcc2

remove conf/druid/single-server/quickstart/_common/historical/jvm.config

342e5d7

Minor changes in python script

6a4a48e

kfaraz added Ease of Use Area - Cli labels Nov 15, 2022

findingrish added 2 commits November 15, 2022 14:38

Add lower bound memory for some services

36b86b6

Additional runtime properties for services

dbfb465

findingrish marked this pull request as ready for review November 15, 2022 15:16

gianm added this to the 25.0 milestone Nov 15, 2022

findingrish added 3 commits November 18, 2022 13:26

Update supervise script to accept command arguments, corresponding ch…

7324252

…anges in druid-quickstart.py

File end newline

72494e9

Limit the ability to start multiple instances of a service, documenta…

e350587

…tion changes

findingrish marked this pull request as draft November 18, 2022 15:12

simplify script arguments

68a1e66

findingrish marked this pull request as ready for review November 18, 2022 19:42

restore changes in medium profile

e570606

kfaraz self-requested a review November 21, 2022 03:37

run-druid refactor

b413d15

gianm reviewed Nov 21, 2022

View reviewed changes

kfaraz reviewed Nov 21, 2022

View reviewed changes

findingrish added 2 commits November 22, 2022 11:09

compute and pass middle manager runtime properties to run-druid

9b8cdc6

supervise script changes to process java opts array use argparse, leave free memory, logging

Remove extra quotes from mm task javaopts array

75d169f

Update single-server.md

a4d14f7

findingrish added 3 commits December 8, 2022 11:24

merge from master

0742c77

Update python.md

41ad0bd

run python3 --version to check if python is installed

08507b4

vogievetsky reviewed Dec 8, 2022

View reviewed changes

vogievetsky approved these changes Dec 8, 2022

View reviewed changes

gianm approved these changes Dec 9, 2022

View reviewed changes

vogievetsky reviewed Dec 9, 2022

View reviewed changes

findingrish added 5 commits December 9, 2022 09:03

merge from master

48ffe3f

Update supervise script

14b8679

start-druid: echo message if python not found

2837dd3

update anchor text

511583e

minor change

d569693

gianm reviewed Dec 9, 2022

View reviewed changes

findingrish added 2 commits December 9, 2022 13:19

Update condition in supervise script

d0e7374

JVM not jvm in docs

e3d3657

vogievetsky merged commit 4ebdfe2 into apache:master Dec 9, 2022

vogievetsky mentioned this pull request Dec 9, 2022

Add zk to the possible service list in the new bin/start-druid script #13543

Closed

gianm mentioned this pull request Dec 10, 2022

Increased druid.worker.capacity in bin/start-druid script #13547

Closed

findingrish mentioned this pull request Dec 12, 2022

[Backport] Druid automated quickstart #13551

Closed

findingrish mentioned this pull request Dec 12, 2022

[Backport] Druid automated quickstart #13552

Merged

kfaraz pushed a commit that referenced this pull request Dec 12, 2022

[Backport] Druid automated quickstart (#13365) (#13552)

5383dc5

Cherry-picked from commit 4ebdfe2

kfaraz mentioned this pull request Dec 15, 2022

Tune the computed value of druid.processing.buffer.sizeBytes #13569

Closed

techdocsmith mentioned this pull request Dec 21, 2022

Remove python installation instructions / update quickstart #13607

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Druid automated quickstart #13365

Druid automated quickstart #13365

findingrish commented Nov 15, 2022 •

edited

Loading

lgtm-com bot commented Nov 15, 2022

lgtm-com bot commented Nov 15, 2022

FrankChen021 commented Nov 17, 2022

findingrish commented Nov 18, 2022

lgtm-com bot commented Nov 18, 2022

lgtm-com bot commented Nov 18, 2022

gianm left a comment •

edited

Loading

lgtm-com bot commented Nov 21, 2022

findingrish commented Nov 21, 2022 •

edited

Loading

kfaraz left a comment

lgtm-com bot commented Dec 8, 2022

lgtm-com bot commented Dec 8, 2022

vogievetsky Dec 8, 2022

vogievetsky Dec 8, 2022

findingrish Dec 9, 2022

vogievetsky left a comment

gianm left a comment

vogievetsky Dec 9, 2022

vogievetsky Dec 9, 2022

findingrish Dec 9, 2022

gianm Dec 9, 2022

findingrish Dec 9, 2022

maytasm commented May 6, 2023 •

edited

Loading

findingrish commented Jul 31, 2023

Druid automated quickstart #13365

Druid automated quickstart #13365

Conversation

findingrish commented Nov 15, 2022 • edited Loading

Description

Usage

Command:

Examples:

Design

Flow

Parameters

Validations

Memory distribution

Running existing profiles

Limitations

Major changes

A note on runtime properties

Release note

lgtm-com bot commented Nov 15, 2022

lgtm-com bot commented Nov 15, 2022

FrankChen021 commented Nov 17, 2022

findingrish commented Nov 18, 2022

lgtm-com bot commented Nov 18, 2022

lgtm-com bot commented Nov 18, 2022

gianm left a comment • edited Loading

Choose a reason for hiding this comment

lgtm-com bot commented Nov 21, 2022

findingrish commented Nov 21, 2022 • edited Loading

kfaraz left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Dec 8, 2022

lgtm-com bot commented Dec 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vogievetsky left a comment

Choose a reason for hiding this comment

gianm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maytasm commented May 6, 2023 • edited Loading

findingrish commented Jul 31, 2023

findingrish commented Nov 15, 2022 •

edited

Loading

gianm left a comment •

edited

Loading

findingrish commented Nov 21, 2022 •

edited

Loading

maytasm commented May 6, 2023 •

edited

Loading