153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

danielealbano · 2022-08-13T19:14:53Z

This PR implements a new number of components with the goal to streamline the development of the Redis module and. at the same time, improve the compatibility and performances.

There are 3 new major components

A python script able to generate from a JSON file a number of headers and c files at runtime, the C files are compiled into a static library which gets injected onto the cachegrand-internal target, the headers are included as needed.
An automated parser able to read the data generated by the scaffolding generator and, in particular the list of arguments, their type and their tokens, and figure out if what is being received is legit and if yes figure out which fields have to be filled out of the command context (e.g., a sort of reflection but almost without overload as the most of the code is pre-generated)
A new hashtable single producer single consumer used to identify the commands requested by the user and to figure out the tokens of the arguments passed in the commands

This PR closes issue #153

…the code under a modules/redis folder

…modules folder

…e related network_protocol_prometheus in module_prometheus

…rk protocols" as they are not just "protocols"

…tication from hello

…ucts to be able to build

… target

…nism to add a command end callback for the commands that aren't implemented

…context" references to use the wording module

…of generalised implementations

…fset tracking for automatic memory freeing

…ed_commands_table.h in 3 separated headers with arguments, key_specs and the info_map

…, handle error messages, handle connection termination and so on)

…onent

…d (invalid received data or no memory), no reason to send a message

…necessary an assert will kick in a few lines before the free

…t parser functions emit errors, instead emit the errors only in the main functions

…set it to null, these kind of arguments can only have tokens

…ages, no need to test the key length per command anymore, etc.)

lgtm-com · 2022-08-14T08:50:13Z

This pull request introduces 1 alert when merging 5398d36 into 8964b24 - view on LGTM.com

new alerts:

1 for Wrong type of arguments to formatting function

…sage_printf_(non)critical functions are checked with the printf arguments validation

…alid

codecov · 2022-08-14T13:10:40Z

Codecov Report

Merging #170 (5521368) into main (8964b24) will increase coverage by 0.43%.
The diff coverage is 77.79%.

@@            Coverage Diff             @@
##             main     #170      +/-   ##
==========================================
+ Coverage   78.04%   78.47%   +0.43%     
==========================================
  Files          95      103       +8     
  Lines        6456     6580     +124     
==========================================
+ Hits         5038     5163     +125     
+ Misses       1418     1417       -1

Impacted Files	Coverage Δ
src/config_cyaml_schema.c	`100.00% <ø> (ø)`
...structures/hashtable/mcmp/hashtable_support_hash.h	`100.00% <ø> (ø)`
...uctures/hashtable/mcmp/hashtable_support_op_arch.c	`93.78% <ø> (ø)`
...c/module/redis/command/module_redis_command_mset.c	`0.00% <0.00%> (ø)`
src/network/channel/network_channel.c	`92.42% <ø> (ø)`
src/network/channel/network_channel_iouring.c	`100.00% <ø> (ø)`
src/network/channel/network_channel_tls.c	`1.24% <ø> (ø)`
src/network/io/network_io_common.c	`94.08% <ø> (ø)`
src/network/io/network_io_common_tls.c	`100.00% <ø> (ø)`
src/network/network.c	`80.00% <0.00%> (+2.22%)`	⬆️
... and 36 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…as part of a token)

danielealbano added 30 commits July 24, 2022 13:08

Add a json containing the specs for the scaffolding to generate

68a9521

Initial draft of the code generator for the Redis commands scaffolding

e5c4280

Separate the struct containing the chunk_info sequence and their count

9cdc235

Rename network_protocol_redis in module_redis in preparation to move …

b2a0324

…the code under a modules/redis folder

Rename network_protocol.c/h in module.c/h and move the files under a …

7011710

…modules folder

Move the redis module under the modules folder

9bd4bbe

Move the prometheus module under the modules folder and rename all th…

b085b20

…e related network_protocol_prometheus in module_prometheus

Rename network_protocols in module_types

db06d30

Reorganize the configuration to move the modules away from the "netwo…

4276938

…rk protocols" as they are not just "protocols"

Update the config file to match the new structure

8518f83

Autogenerated commands never define command end

8e390e4

Add initial code generation for the callbacks

7cc0796

Add required headers for the auto generated commands callbacks

4bacda7

Fix the basic auto generated command free

52d46d3

The command table is now auto generated, drop the hardcoded one

fe7da94

Can't support sub arguments of sub arguments for now, drop the authen…

454cf97

…tication from hello

Import the auto generated headers as needed and add the necessary str…

878e0d4

…ucts to be able to build

Add the automatic generation of the scaffolding to cmake via a custom…

f857b50

… target

Plenty of small fixes to the scaffolding auto generation, add a mecha…

c1de375

…nism to add a command end callback for the commands that aren't implemented

Various improvements, add prep info for the parsing of the incoming data

30b3037

Rename modules in module and all the "network_protocol" or "protocol_…

c2ea974

…context" references to use the wording module

Callbacks for the command begin and free have been dropped in favour …

060c325

…of generalised implementations

Fix arguments, sub arguments and context generation and add fields of…

e0b2443

…fset tracking for automatic memory freeing

Fix headers for cpp tests

3fb4cd1

Refactor the scaffolding generator to spllit module_redis_autogenerat…

ed52dde

…ed_commands_table.h in 3 separated headers with arguments, key_specs and the info_map

Add a component to handle the redis connections (send out ok messages…

88af86e

…, handle error messages, handle connection termination and so on)

Refactor module_redis to take advantage of module_redis_context

8d25d61

C++ doesn't allow to OR plain ENUM values, add a wrapper for it

2bec0ec

Initial implementation of the tests for the module_redis_command comp…

73ed14b

…onent

Style fixes for the scaffolding generator

c909277

danielealbano self-assigned this Aug 13, 2022

danielealbano added this to In Progress in cachegrand via automation Aug 13, 2022

danielealbano linked an issue Aug 13, 2022 that may be closed by this pull request

Implement a code generator to to automatically create all redis commands scaffolding, arguments parsing and commands callbacks #153

Closed

danielealbano added 10 commits August 14, 2022 10:38

If the command parser returns false the connection needs to be aborte…

9645363

…d (invalid received data or no memory), no reason to send a message

If there is a parsing error a message can be sent back

a0d0364

Fix various error messages

90cbd85

If an error message is set, no failure should be reported as if it's …

5bb1b08

…necessary an assert will kick in a few lines before the free

To make the code more readable, avoid that the nested command argumen…

b062d91

…t parser functions emit errors, instead emit the errors only in the main functions

One more error message cleanup

e5b427a

Style fixes

402f9ff

In argument end, if a ONEOF argument type is supposed to be expected …

8edd4a6

…set it to null, these kind of arguments can only have tokens

Fix tests for unfinished testing unit

7a9fab6

Fix tests to work with the new automated parser (different error mess…

5398d36

…ages, no need to test the key length per command anymore, etc.)

danielealbano added 5 commits August 14, 2022 12:10

Fix LGTM alerts and ensure that the module_redis_connection_error_mes…

9a72133

…sage_printf_(non)critical functions are checked with the printf arguments validation

Fix fixtures used in config related tests

745bdaa

Add an extra test to ensure that etc/cachegrand.yaml.skel is always v…

cc96f4b

…alid

Style fix

72f548d

Build the path to the skel config file dinamically

c5417ae

danielealbano added 6 commits August 15, 2022 15:47

Ensure ping command is able to send out long replies

8110106

Handle correctly 0 length arguments

e73c983

Update not implemented test

117eed5

Fix test for RESP command with one argument with 0 length

d1b039f

Add a test to validate the key length when it's non positional (e.g. …

7c82e64

…as part of a token)

Add codecov.yml to disable codecov/status

5521368

danielealbano marked this pull request as ready for review August 15, 2022 14:22

danielealbano merged commit 0edd630 into main Aug 15, 2022

cachegrand automation moved this from In Progress to Completed Aug 15, 2022

danielealbano deleted the 153-implement-a-code-generator-to-to-automatically-create-all-redis-commands-scaffolding-arguments-parsing-and-commands-callbacks branch August 15, 2022 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

danielealbano commented Aug 13, 2022

lgtm-com bot commented Aug 14, 2022

codecov bot commented Aug 14, 2022 •

edited

Loading

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

Conversation

danielealbano commented Aug 13, 2022

lgtm-com bot commented Aug 14, 2022

codecov bot commented Aug 14, 2022 • edited Loading

Codecov Report

codecov bot commented Aug 14, 2022 •

edited

Loading