Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

Merged
merged 114 commits into from
Aug 15, 2022
Merged

153 implement a code generator to to automatically create all redis commands scaffolding arguments parsing and commands callbacks #170

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
68a9521
Add a json containing the specs for the scaffolding to generate
danielealbano Jul 24, 2022
e5c4280
Initial draft of the code generator for the Redis commands scaffolding
danielealbano Jul 24, 2022
9cdc235
Separate the struct containing the chunk_info sequence and their count
danielealbano Jul 24, 2022
b2a0324
Rename network_protocol_redis in module_redis in preparation to move …
danielealbano Jul 24, 2022
7011710
Rename network_protocol.c/h in module.c/h and move the files under a …
danielealbano Jul 24, 2022
9bd4bbe
Move the redis module under the modules folder
danielealbano Jul 24, 2022
b085b20
Move the prometheus module under the modules folder and rename all th…
danielealbano Jul 24, 2022
db06d30
Rename network_protocols in module_types
danielealbano Jul 24, 2022
4276938
Reorganize the configuration to move the modules away from the "netwo…
danielealbano Jul 24, 2022
8518f83
Update the config file to match the new structure
danielealbano Jul 24, 2022
8e390e4
Autogenerated commands never define command end
danielealbano Jul 24, 2022
7cc0796
Add initial code generation for the callbacks
danielealbano Jul 24, 2022
4bacda7
Add required headers for the auto generated commands callbacks
danielealbano Jul 24, 2022
52d46d3
Fix the basic auto generated command free
danielealbano Jul 24, 2022
fe7da94
The command table is now auto generated, drop the hardcoded one
danielealbano Jul 24, 2022
454cf97
Can't support sub arguments of sub arguments for now, drop the authen…
danielealbano Jul 24, 2022
878e0d4
Import the auto generated headers as needed and add the necessary str…
danielealbano Jul 24, 2022
f857b50
Add the automatic generation of the scaffolding to cmake via a custom…
danielealbano Jul 24, 2022
c1de375
Plenty of small fixes to the scaffolding auto generation, add a mecha…
danielealbano Jul 24, 2022
30b3037
Various improvements, add prep info for the parsing of the incoming data
danielealbano Jul 24, 2022
c2ea974
Rename modules in module and all the "network_protocol" or "protocol_…
danielealbano Jul 25, 2022
060c325
Callbacks for the command begin and free have been dropped in favour …
danielealbano Jul 25, 2022
e0b2443
Fix arguments, sub arguments and context generation and add fields of…
danielealbano Jul 25, 2022
3fb4cd1
Fix headers for cpp tests
danielealbano Jul 25, 2022
ed52dde
Refactor the scaffolding generator to spllit module_redis_autogenerat…
danielealbano Jul 28, 2022
88af86e
Add a component to handle the redis connections (send out ok messages…
danielealbano Jul 28, 2022
8d25d61
Refactor module_redis to take advantage of module_redis_context
danielealbano Jul 28, 2022
2bec0ec
C++ doesn't allow to OR plain ENUM values, add a wrapper for it
danielealbano Jul 28, 2022
73ed14b
Initial implementation of the tests for the module_redis_command comp…
danielealbano Jul 28, 2022
c909277
Style fixes for the scaffolding generator
danielealbano Jul 30, 2022
8ebbec2
Drop the generation of the callbacks for argument_stream_begin/data/e…
danielealbano Jul 30, 2022
08d6238
use uint64_t for unix timestamps to handle milliseconds properly
danielealbano Jul 30, 2022
2573a8d
Drop the code to generate the not required anymore callbacks (part 2)
danielealbano Jul 30, 2022
41c9075
Cleanup the autogenerated contexts
danielealbano Jul 30, 2022
ae90641
Correct the argument size calculation, add sub arguments, sub argumen…
danielealbano Jul 30, 2022
b9fdb12
Add automatic code generation to free the generated command contexts
danielealbano Jul 30, 2022
5fa66d0
Fix the command info table generation and add a flag to indicate if t…
danielealbano Jul 30, 2022
07fe7a6
Refactor the storage db to use sequence of chunks instead of having 2…
danielealbano Jul 30, 2022
66213bb
nit: In the slab allocator, if asked to allocate 0 byte return null
danielealbano Jul 30, 2022
2db2d9d
Update cmake to indicate all the files generated by the scaffolding g…
danielealbano Jul 30, 2022
e66900f
Add a macro to get the size of a member of a struct without having an…
danielealbano Jul 30, 2022
926c93a
Add the storage db to the connection context
danielealbano Jul 30, 2022
254f2bd
Restore the callback to free the command context as the auto generate…
danielealbano Jul 30, 2022
b62e8e7
Drop the unused callbacks and update the command info struct to match…
danielealbano Jul 30, 2022
c867b80
Update the argument struct to match the updated autogenerated one
danielealbano Jul 30, 2022
3068962
New command argument parser struct, added it to the connection contex…
danielealbano Jul 30, 2022
3318586
Always fill up the command argument parser context with zeros when th…
danielealbano Jul 30, 2022
0767f1b
Stub functions for the callbacks to handle the command arguments
danielealbano Jul 30, 2022
44f6228
Introduction of a function to build a token map, temporary, the map h…
danielealbano Jul 30, 2022
c24f182
Fix various bugs in the automatic dump of the command context
danielealbano Jul 30, 2022
975552a
Update module redis command
danielealbano Jul 30, 2022
baa5b3c
Pass the storage db onto the connection context, drop the callbacks f…
danielealbano Jul 30, 2022
91eb61c
Update test after renaming function
danielealbano Jul 30, 2022
dc31ec2
Update commands to include the amount of required arguments, if they …
danielealbano Jul 30, 2022
ff40ab5
Add some macro to catch errors at compile time
danielealbano Aug 8, 2022
4b13e75
Drop the key specs as they are unused but keep in the json as took qu…
danielealbano Aug 8, 2022
c1ba87e
Fix a bug in the slab allocator mem realloc which was preventing the …
danielealbano Aug 8, 2022
b084e59
Add some new wrapper functions to avoid code duplication in the stora…
danielealbano Aug 8, 2022
7c7619f
Reorganize module_redis_command_info to optimize memory consumption a…
danielealbano Aug 8, 2022
e18a9cd
Fix the argument processing flow, argument being has to be called bef…
danielealbano Aug 8, 2022
b62bc83
Drop command length, use string_len instead
danielealbano Aug 8, 2022
103133f
Improve automatic argument parsing implementation, still doesn't hand…
danielealbano Aug 8, 2022
f418a92
Minor performance optimization
danielealbano Aug 8, 2022
739c90b
Re-implement all the commands leveraging the automated argument parser
danielealbano Aug 8, 2022
7845f0f
key_specs dropped
danielealbano Aug 8, 2022
85bc6d7
Add support for long & short strings, differentiate in commands.json …
danielealbano Aug 8, 2022
0745df1
Minor headers cleanup
danielealbano Aug 8, 2022
4fff9dd
Rename the benchmarks and the tests for the multi producer multi cons…
danielealbano Aug 11, 2022
6236f90
Fix typo
danielealbano Aug 11, 2022
3094efe
Implement a new hashtable single producer single consumer used intern…
danielealbano Aug 11, 2022
99431d3
Add the hash fnv1 used by hashtable spsc
danielealbano Aug 11, 2022
346d380
Switch some functions to static always inline
danielealbano Aug 11, 2022
7e04b3e
Add tests for the hashtable single producer single consumer
danielealbano Aug 11, 2022
d395282
Fix headers
danielealbano Aug 11, 2022
9c7d9ce
Add a flag to enable strict parsing of the redis commands
danielealbano Aug 11, 2022
ddb3074
Update the configuration documentation after the massive reorganisati…
danielealbano Aug 11, 2022
1d87974
Benchmarks should be compiled with -O3
danielealbano Aug 11, 2022
81a6bb1
Finish the initial cut implementation of the redis commands parser
danielealbano Aug 11, 2022
406d186
No reason to generate "values" for bool arguments because they are al…
danielealbano Aug 11, 2022
64b6c3c
Fix message
danielealbano Aug 11, 2022
4104553
Move the hashtable mpmc tests under the data_structures folder
danielealbano Aug 12, 2022
050b1bf
Add a required include to the c files generated by the scaffolding ge…
danielealbano Aug 12, 2022
a43ebf8
Move some general structs from protocol redis reader to protocol redis
danielealbano Aug 12, 2022
7dc362d
Add stddef to some additional files
danielealbano Aug 12, 2022
c432aba
Refactor the hashtable spsc to make it more testable, also add hashta…
danielealbano Aug 12, 2022
cb1f956
Add a new component to pre-generation at the start of some data for t…
danielealbano Aug 12, 2022
2ed41c9
Fix headers order
danielealbano Aug 12, 2022
b1251f0
Remove the token found from token entry and get the live value from t…
danielealbano Aug 12, 2022
c873199
General performances improvements
danielealbano Aug 12, 2022
4a6f7a8
Search the command using the pre-generated hashtable for the commands…
danielealbano Aug 12, 2022
2baf697
Fix commented out test to fix tests build
danielealbano Aug 12, 2022
42b6579
Merge branch 'main' into 153-implement-a-code-generator-to-to-automat…
danielealbano Aug 12, 2022
2d6043b
Various fix ups after merging main onto the branch
danielealbano Aug 12, 2022
9645363
If the command parser returns false the connection needs to be aborte…
danielealbano Aug 14, 2022
a0d0364
If there is a parsing error a message can be sent back
danielealbano Aug 14, 2022
90cbd85
Fix various error messages
danielealbano Aug 14, 2022
5bb1b08
If an error message is set, no failure should be reported as if it's …
danielealbano Aug 14, 2022
b062d91
To make the code more readable, avoid that the nested command argumen…
danielealbano Aug 14, 2022
e5b427a
One more error message cleanup
danielealbano Aug 14, 2022
402f9ff
Style fixes
danielealbano Aug 14, 2022
8edd4a6
In argument end, if a ONEOF argument type is supposed to be expected …
danielealbano Aug 14, 2022
7a9fab6
Fix tests for unfinished testing unit
danielealbano Aug 14, 2022
5398d36
Fix tests to work with the new automated parser (different error mess…
danielealbano Aug 14, 2022
9a72133
Fix LGTM alerts and ensure that the module_redis_connection_error_mes…
danielealbano Aug 14, 2022
745bdaa
Fix fixtures used in config related tests
danielealbano Aug 14, 2022
cc96f4b
Add an extra test to ensure that etc/cachegrand.yaml.skel is always v…
danielealbano Aug 14, 2022
72f548d
Style fix
danielealbano Aug 14, 2022
c5417ae
Build the path to the skel config file dinamically
danielealbano Aug 14, 2022
8110106
Ensure ping command is able to send out long replies
danielealbano Aug 15, 2022
e73c983
Handle correctly 0 length arguments
danielealbano Aug 15, 2022
117eed5
Update not implemented test
danielealbano Aug 15, 2022
d1b039f
Fix test for RESP command with one argument with 0 length
danielealbano Aug 15, 2022
7c82e64
Add a test to validate the key length when it's non positional (e.g. …
danielealbano Aug 15, 2022
5521368
Add codecov.yml to disable codecov/status
danielealbano Aug 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Various improvements, add prep info for the parsing of the incoming data
  • Loading branch information
@danielealbano
danielealbano committed Jul 24, 2022
commit 30b3037adfe3f7d7eca5eaf6ac7e134a6e9cab2a
6 changes: 3 additions & 3 deletions src/modules/redis/module_redis.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@

#include "module_redis.h"
#include "module_redis_autogenerated_commands_callbacks.h"
#include "module_redis_commands_table_autogenerated.h"
#include "module_redis_autogenerated_commands_table.h"

#define TAG "module_redis"

Expand Down Expand Up @@ -312,7 +312,7 @@ bool module_redis_process_data(
goto end;
}
} else {
if (protocol_context->command_info->required_positional_arguments_count >
if (protocol_context->command_info->required_arguments_count >
reader_context->arguments.count - 1) {
protocol_context->skip_command = true;
continue;
Expand Down Expand Up @@ -491,7 +491,7 @@ bool module_redis_process_data(

assert(protocol_context->command_info != NULL);

if (protocol_context->command_info->required_positional_arguments_count >
if (protocol_context->command_info->required_arguments_count >
reader_context->arguments.count - 1) {
send_buffer = send_buffer_start = network_send_buffer_acquire_slice(channel, slice_length);
if (send_buffer_start == NULL) {
Expand Down
67 changes: 22 additions & 45 deletions src/modules/redis/module_redis.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,24 +140,23 @@ extern "C" {
argument_full, \
MODULE_REDIS_COMMAND_FUNCPTR_ARGUMENTS_ARGUMENT_FULL)

#define MODULE_REDIS_COMMAND_FUNCPTR_COMMAND_END_AUTOGEN(COMMAND_FUNC_PTR) \
MODULE_REDIS_COMMAND_FUNCPTR_GENERIC_AUTOGEN( \
COMMAND_FUNC_PTR, \
command_end, \
MODULE_REDIS_COMMAND_FUNCPTR_ARGUMENTS_COMMAND_END)

#define MODULE_REDIS_COMMAND_FUNCPTR_COMMAND_FREE_AUTOGEN(COMMAND_FUNC_PTR) \
MODULE_REDIS_COMMAND_FUNCPTR_GENERIC_AUTOGEN( \
COMMAND_FUNC_PTR, \
command_free, \
MODULE_REDIS_COMMAND_FUNCPTR_ARGUMENTS_COMMAND_END)

#define MODULE_REDIS_COMMAND_AUTOGEN(ID, COMMAND, COMMAND_FUNC_PTR, POS_ARGS_COUNT) \
#define MODULE_REDIS_COMMAND_AUTOGEN(ID, COMMAND, COMMAND_FUNC_PTR, REQUIRED_ARGS_COUNT, KEY_SPECS_COUNT, ARGS_COUNT) \
{ \
.command = MODULE_REDIS_COMMAND_##ID, \
.length = sizeof(COMMAND) - 1, /* sizeof takes into account the NULL char at the end, different behaviour than strlen */ \
.string = (COMMAND), \
.string_len = strlen(COMMAND), \
.context_struct_size = sizeof(CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), context_t)), \
.key_specs = CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), key_specs), \
.key_specs_count = (ARGS_COUNT), \
.arguments = CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), arguments), \
.arguments_count = (ARGS_COUNT), \
.command_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME_AUTOGEN(COMMAND_FUNC_PTR, command_begin), \
.argument_require_stream_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME_AUTOGEN(COMMAND_FUNC_PTR, argument_require_stream), \
.argument_stream_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME_AUTOGEN(COMMAND_FUNC_PTR, argument_stream_begin), \
Expand All @@ -166,15 +165,20 @@ extern "C" {
.argument_full_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME_AUTOGEN(COMMAND_FUNC_PTR, argument_full), \
.command_end_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_end), \
.command_free_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME_AUTOGEN(COMMAND_FUNC_PTR, command_free), \
.required_positional_arguments_count = (POS_ARGS_COUNT) \
.required_arguments_count = (REQUIRED_ARGS_COUNT) \
}

#define MODULE_REDIS_COMMAND(ID, COMMAND, COMMAND_FUNC_PTR, POS_ARGS_COUNT) \
#define MODULE_REDIS_COMMAND(ID, COMMAND, COMMAND_FUNC_PTR, REQUIRED_ARGS_COUNT, KEY_SPECS_COUNT, ARGS_COUNT) \
{ \
.command = MODULE_REDIS_COMMAND_##ID, \
.length = sizeof(COMMAND) - 1, /* sizeof takes into account the NULL char at the end, different behaviour than strlen */ \
.string = (COMMAND), \
.string_len = strlen(COMMAND), \
.context_struct_size = sizeof(CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), context_t)), \
.key_specs = CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), key_specs), \
.key_specs_count = (ARGS_COUNT), \
.arguments = CONCAT(CONCAT(module_redis_command, COMMAND_FUNC_PTR), arguments), \
.arguments_count = (ARGS_COUNT), \
.command_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_begin), \
.argument_require_stream_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, argument_require_stream), \
.argument_stream_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, argument_stream_begin), \
Expand All @@ -183,41 +187,7 @@ extern "C" {
.argument_full_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, argument_full), \
.command_end_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_end), \
.command_free_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_free), \
.required_positional_arguments_count = (POS_ARGS_COUNT) \
}

#define MODULE_REDIS_COMMAND_NO_STREAM(ID, COMMAND, COMMAND_FUNC_PTR, POS_ARGS_COUNT) \
{ \
.command = MODULE_REDIS_COMMAND_##ID, \
.length = sizeof(COMMAND) - 1, /* sizeof takes into account the NULL char at the end, different behaviour than strlen */ \
.string = (COMMAND), \
.string_len = strlen(COMMAND), \
.command_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_begin), \
.argument_require_stream_funcptr = NULL, \
.argument_stream_begin_funcptr = NULL, \
.argument_stream_data_funcptr = NULL, \
.argument_stream_end_funcptr = NULL, \
.argument_full_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, argument_full), \
.command_end_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_end), \
.command_free_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_free), \
.required_positional_arguments_count = (POS_ARGS_COUNT) \
}

#define MODULE_REDIS_COMMAND_NO_ARGS(ID, COMMAND, COMMAND_FUNC_PTR, POS_ARGS_COUNT) \
{ \
.command = MODULE_REDIS_COMMAND_##ID, \
.length = sizeof(COMMAND) - 1, /* sizeof takes into account the NULL char at the end, different behaviour than strlen */ \
.string = (COMMAND), \
.string_len = strlen(COMMAND), \
.command_begin_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_begin), \
.argument_require_stream_funcptr = NULL, \
.argument_stream_begin_funcptr = NULL, \
.argument_stream_data_funcptr = NULL, \
.argument_stream_end_funcptr = NULL, \
.argument_full_funcptr = NULL, \
.command_end_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_end), \
.command_free_funcptr = MODULE_REDIS_COMMAND_FUNCPTR_NAME(COMMAND_FUNC_PTR, command_free), \
.required_positional_arguments_count = (POS_ARGS_COUNT) \
.required_arguments_count = (REQUIRED_ARGS_COUNT) \
}

typedef void module_redis_command_context_t;
Expand Down Expand Up @@ -319,6 +289,11 @@ struct module_redis_command_info {
// Redis longest command is 10 chars
char string[32];
uint8_t string_len;
uint16_t context_struct_size;
module_redis_command_key_spec_t *key_specs;
uint16_t key_specs_count;
module_redis_command_argument_t *arguments;
uint16_t arguments_count;
module_redis_command_begin_funcptr_t *command_begin_funcptr;
module_redis_command_argument_require_stream_funcptr_t *argument_require_stream_funcptr;
module_redis_command_argument_stream_begin_funcptr_t *argument_stream_begin_funcptr;
Expand All @@ -327,7 +302,7 @@ struct module_redis_command_info {
module_redis_command_argument_full_funcptr_t *argument_full_funcptr;
module_redis_command_end_funcptr_t *command_end_funcptr;
module_redis_command_free_funcptr_t *command_free_funcptr;
uint8_t required_positional_arguments_count;
uint8_t required_arguments_count;
};

struct module_redis_context {
Expand All @@ -341,6 +316,8 @@ struct module_redis_context {
bool skip_command;
};

#include "module_redis_autogenerated_commands_contexts.h"

void module_redis_accept(
network_channel_t *channel);

Expand Down
Loading