Assign the `_lpython_argv()` value to a global `list[str]` variable #1679

Thirumalai-Shaktivel · 2023-04-07T11:27:36Z

Related: #1358

Thirumalai-Shaktivel · 2023-04-07T11:28:52Z

@Smit-create Does CPP support Lists?

I get Seg_fault for the file: loop2.py in the CPP backend

Smit-create · 2023-04-08T18:22:13Z

@Smit-create Does CPP support Lists?

CPP has many missing things. I would suggest just using C for now. Extending it to CPP won't be harder.

certik · 2023-04-08T19:20:15Z

Yes, let's use C for now.

Thirumalai-Shaktivel · 2023-04-09T06:42:44Z

I get the following error:

$ lpython examples/expr2.py --backend=c
Undefined symbols for architecture arm64:
  "__lfortran_strcpy", referenced from:
      _list_append_str in expr2__tmp__generated__-02c27a.o
      _list_insert_str in expr2__tmp__generated__-02c27a.o
  "__lpython_get_argc", referenced from:
      __lpython_argv in expr2__tmp__generated__-02c27a.o
  "__lpython_get_argv", referenced from:
      __lpython_argv in expr2__tmp__generated__-02c27a.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
The command 'gcc -o expr2.out expr2__tmp__generated__.c  -I /Users/thirumalai/Open_Source/lpython/src/bin/../libasr/runtime' failed.
sh: ./expr2.out: No such file or directory

We seem to specify the correct path for lfortran_intrinsics, but we still get the undefined symbol. What might be the reason?

Thirumalai-Shaktivel · 2023-04-09T07:34:43Z

integration_tests/CMakeLists.txt

@@ -219,8 +219,8 @@ RUN(NAME bindc_01            LABELS cpython llvm c)
 RUN(NAME bindc_02            LABELS cpython llvm c)
 RUN(NAME bindc_04            LABELS llvm c)
 RUN(NAME bindc_07            LABELS cpython llvm c)
-RUN(NAME exit_01             LABELS cpython llvm c wasm wasm_x86 wasm_x64)
-RUN(NAME exit_02     FAIL    LABELS cpython llvm c wasm wasm_x86 wasm_x64)


from sys import exit

This also imports the argv initialization.
As Wasm doesn't support the list type yet, I removed it from here.

I think you should not de-register test for certain backends. That would be wrong. Instead create a new one and register it with limited backends.

The problem is exit_01.py contains from sys import exit.
Module sys contains some list operation in the global scope that doesn't work in WASM.
I think we have to wait until Lists are implemented in WASM.
@czgdp1807 What do you say?

I wouldn't de-register tests. @Shaikh-Ubaid How long would it take to support lists in WASM? Is it even possible to support lists in WASM?

@Shaikh-Ubaid How long would it take to support lists in WASM? Is it even possible to support lists in WASM?

I think supporting lists have some prerequisites like structs/pointers. Thus, I think it could take sometime to reach lists.

Is it even possible to support lists in WASM?

Yes, I think it should be possible to support lists in the wasm backend.

@Thirumalai-Shaktivel Then I would say extract out the parts in exit_01.py which are important for LLVM testing and then put it into a new file exit_0n.py (don't put sys module imports and register this new file with wasm, wasm_x86, wasm_x64. Then you can remove WASM tests from exit_01.py. Same for #1679 (comment)

Thirumalai-Shaktivel · 2023-04-10T04:59:05Z

I have a linker issue in the C backend, can anyone please provide suggestions on this?
#1679 (comment)

Thirumalai-Shaktivel · 2023-04-10T05:44:35Z

I fixed this by specifying the lpython_runtime path

gcc -o expr2.out expr2__tmp__generated__.c  -I /Users/thirumalai/Open_Source/lpython/src/bin/../libasr/runtime -L"/Users/thirumalai/Open_Source/lpython/src/bin/../runtime" -Wl,-rpath,"/Users/thirumalai/Open_Source/lpython/src/bin/../runtime" -llpython_runtime -lm

@certik let me know if any other options are to be specified here.

Thirumalai-Shaktivel · 2023-04-10T09:02:47Z

The following C code causes seg fault, which is the reason for the failure in this PR,
@Smit-create see if you can find any bugs at the first glance. I will try to debug it as well.
I went through the code, and it seems there is no issue with this code. I suspect the issue is related to char * allocation or something.

#include <inttypes.h>

#include <stdlib.h>
#include <stdbool.h>
#include <stdio.h>
#include <string.h>
#include <lfortran_intrinsics.h>

#define ASSERT(cond)                                                           \
    {                                                                          \
        if (!(cond)) {                                                         \
            printf("%s%s", "ASSERT failed: ", __FILE__);                       \
            printf("%s%s", "\nfunction ", __func__);                           \
            printf("%s%d%s", "(), line number ", __LINE__, " at \n");          \
            printf("%s%s", #cond, "\n");                                       \
            exit(1);                                                           \
        }                                                                      \
    }
#define ASSERT_MSG(cond, msg)                                                  \
    {                                                                          \
        if (!(cond)) {                                                         \
            printf("%s%s", "ASSERT failed: ", __FILE__);                       \
            printf("%s%s", "\nfunction ", __func__);                           \
            printf("%s%d%s", "(), line number ", __LINE__, " at \n");          \
            printf("%s%s", #cond, "\n");                                       \
            printf("%s", "ERROR MESSAGE:\n");                                  \
            printf("%s%s", msg, "\n");                                         \
            exit(1);                                                           \
        }                                                                      \
    }


struct dimension_descriptor
{
    int32_t lower_bound, length;
};

struct list_str {
    int32_t capacity;
    int32_t current_end_point;
    char** data;
};

inline bool compare_list_str_(struct list_str a, struct list_str b);
inline bool compare_str(char* a, char* b);
inline void print_list_str_(struct list_str a);
inline void print_str(char* a);
inline void list_init_str(struct list_str* x, int32_t capacity);
inline void list_deepcopy_str(struct list_str* src, struct list_str* dest);
inline void resize_if_needed_str(struct list_str* x);
inline void list_append_str(struct list_str* x, char* element);
inline void list_insert_str(struct list_str* x, int pos, char* element);
inline int list_find_item_str(struct list_str* x, char* element);
inline void list_remove_str(struct list_str* x, char* element);
inline void list_clear_str(struct list_str* x);
inline struct list_str* list_concat_str(struct list_str* left, struct list_str* right);
inline struct list_str* list_section_str(struct list_str* x, int32_t idx1, int32_t idx2, int32_t step, bool i1_present, bool i2_present);


// Implementations
struct list_str argv;
int32_t _lpython_get_argc();

char* _lpython_get_argv(int32_t index);

struct list_str _lpython_argv()
{
    struct list_str _lpython_return_variable;
    int32_t argc;
    struct list_str argv;
    int32_t i;
    argc = _lpython_get_argc();
    for (i=0; i<=argc - 1; i++) {
        list_append_str(&argv, _lpython_get_argv(i));
    }
    list_deepcopy_str(&argv, &_lpython_return_variable);

    return _lpython_return_variable;
}

void global_initializer()
{
    struct list_str constname0 = _lpython_argv();
    list_deepcopy_str(&constname0, &argv);

}

void main0()
{
    struct list_str x;
    list_deepcopy_str(&argv, &x);

    printf("");
    print_list_str_(x);
    printf("\n");
}

void _lpython_main_program()
{
    global_initializer();
    main0();
}

int main(int argc, char* argv[])
{
    _lpython_main_program();
    return 0;
}

bool compare_str(char* a, char* b) {
    return strcmp(a, b) == 0;
}

bool compare_list_str_(struct list_str a, struct list_str b) {
    if (a.current_end_point != b.current_end_point)
        return false;
    for (int i=0; i<a.current_end_point; i++) {
        if (!compare_str(a.data[i], b.data[i]))
            return false;
    }
    return true;
}

void print_str(char* a) {
    printf("'%s'", a);
}

void print_list_str_(struct list_str a) {
    printf("[");
    for (int i=0; i<a.current_end_point; i++) {
        print_str(a.data[i]);
        if (i+1!=a.current_end_point)
            printf(", ");
    }
    printf("]");
}

void list_init_str(struct list_str* x, int32_t capacity) {
    x->capacity = capacity;
    x->current_end_point = 0;
    x->data = (char**) malloc(capacity * sizeof(char*));
}

void list_deepcopy_str(struct list_str* src, struct list_str* dest) {
    dest->capacity = src->capacity;
    dest->current_end_point = src->current_end_point;
    dest->data = (char**) malloc(src->capacity * sizeof(char*));
    memcpy(dest->data, src->data, src->capacity * sizeof(char*));      // Seg-fault seems to happen in this function
}

void resize_if_needed_str(struct list_str* x) {
    if (x->capacity == x->current_end_point) {
        x->capacity = 2 * x->capacity + 1;
        x->data = (char**) realloc(x->data, x->capacity * sizeof(char*));
    }
}

void list_append_str(struct list_str* x, char* element) {
    resize_if_needed_str(x);
    x->data[x->current_end_point] = NULL;
    _lfortran_strcpy(&x->data[x->current_end_point], element);
    x->current_end_point += 1;
}

void list_insert_str(struct list_str* x, int pos, char* element) {
    resize_if_needed_str(x);
    int pos_ptr = pos;
    char* tmp_ptr = x->data[pos];
    char* tmp;
    while (x->current_end_point > pos_ptr) {
        tmp = x->data[pos_ptr + 1];
        x->data[pos_ptr + 1] = tmp_ptr;
        tmp_ptr = tmp;
        pos_ptr++;
    }

    x->data[pos] = NULL;
    _lfortran_strcpy(&x->data[pos], element);
    x->current_end_point += 1;
}

int list_find_item_str(struct list_str* x, char* element) {
    int el_pos = 0;
    while (x->current_end_point > el_pos) {
        if (compare_str(x->data[el_pos], element)) return el_pos;
        el_pos++;
    }
    return -1;
}

void list_remove_str(struct list_str* x, char* element) {
    int el_pos = list_find_item_str(x, element);
    while (x->current_end_point > el_pos) {
        int tmp = el_pos + 1;
        x->data[el_pos] = x->data[tmp];
        el_pos = tmp;
    }
    x->current_end_point -= 1;
}

void list_clear_str(struct list_str* x) {
    free(x->data);
    x->capacity = 4;
    x->current_end_point = 0;
    x->data = (char**) malloc(x->capacity * sizeof(char*));
}

struct list_str* list_concat_str(struct list_str* left, struct list_str* right) {
    struct list_str *result = (struct list_str*)malloc(sizeof(struct list_str));
    list_init_str(result, left->current_end_point + right->current_end_point);
    memcpy(result->data, left->data, left->current_end_point * sizeof(char*));
    memcpy(result->data + left->current_end_point, right->data, right->current_end_point * sizeof(char*));
    result->current_end_point = left->current_end_point + right->current_end_point;
    return result;
}

struct list_str* list_section_str(struct list_str* x, int32_t idx1, int32_t idx2, int32_t step, bool i1_present, bool i2_present) {
    int s_len = x->current_end_point;
    if (step == 0) {
        printf("slice step cannot be zero");
        exit(1);
    }
    idx1 = idx1 < 0 ? idx1 + s_len : idx1;
    idx2 = idx2 < 0 ? idx2 + s_len : idx2;
    idx1 = i1_present ? idx1 : (step > 0 ? 0 : s_len-1);
    idx2 = i2_present ? idx2 : (step > 0 ? s_len : -1);
    idx2 = step > 0 ? (idx2 > s_len ? s_len : idx2) : idx2;
    idx1 = step < 0 ? (idx1 >= s_len ? s_len-1 : idx1) : idx1;
    struct list_str *__tmp = (struct list_str*) malloc(sizeof(struct list_str));
    list_init_str(__tmp, 4);
    int s_i = idx1;
    while((step > 0 && s_i >= idx1 && s_i < idx2) ||
        (step < 0 && s_i <= idx1 && s_i > idx2)) {
    list_append_str(__tmp, x->data[s_i]);
    s_i+=step;
    }
    return __tmp;
}

Thirumalai-Shaktivel · 2023-04-10T09:20:00Z

I was able to debug the issue!

int main(int argc, char* argv[])
{
    _lpython_main_program();
    return 0;
}

I think we forgot to set the argc and argv values here!

Thirumalai-Shaktivel · 2023-04-10T11:22:29Z

After calling the function:_lpython_set_argv(), I still get seg_fault. We have to debug the issue. (P.S. If I use a printf(" ") in c code, the seg_fault is gone. I wonder why?)

int main(int argc, char* argv[])
{
    _lpython_set_argv(argc, argv);
    _lpython_main_program();
    return 0;
}

certik · 2023-04-11T00:39:28Z

src/runtime/sys.py

@@ -7,8 +7,16 @@ def exit(error_code: i32):

    quit(error_code)

+#  >>> argv >>>


I would use different signs than <<< and >>> because those are used for git conflict resolution, so it might be confusing.

certik · 2023-04-11T00:40:12Z

It needs to be debugged, don't know what the issue is quickly.

certik · 2023-04-12T16:32:18Z

The struct just has to be initialized, so right after:

    struct list_str argv;

do

    argv.capacity = 0;                                                       
    argv.current_end_point = 0;                            
    argv.data = NULL;

certik · 2023-04-12T16:35:19Z

A fix might be possible to manually initialize the list in the Python code. This might be enough to get this PR working.

However, let's also create a minimal reproducible example of this list bug, and report it into an issue, as LPython either has to give a compiler time error, or a runtime error, but it should not segfault.

Thirumalai-Shaktivel · 2023-04-13T05:15:19Z

bindc_03 failed in the C backend: https://github.com/lcompilers/lpython/actions/runs/4685562144/jobs/8302777100.
How do I compile bindc_03.py by hand? (It seems we have to compile a C file (bindc_03b.c) and link it)
@czgdp1807

Thirumalai-Shaktivel · 2023-04-18T10:18:45Z

I get:

$ lpython integration_tests/bindc_03.py --backend c
/usr/bin/ld: /tmp/ccROlo4S.o: in function `gpy':
bindc_03__tmp__generated__.c:(.text+0xd1): undefined reference to `g'
/usr/bin/ld: /tmp/ccROlo4S.o: in function `run':
bindc_03__tmp__generated__.c:(.text+0x523): undefined reference to `get_array'
collect2: error: ld returned 1 exit status
The command 'gcc -o bindc_03.out bindc_03__tmp__generated__.c  -I /home/thirumalai/Open_Source/lpython/src/bin/../libasr/runtime -L"/home/thirumalai/Open_Source/lpython/src/bin/../runtime" -Wl,-rpath,"/home/thirumalai/Open_Source/lpython/src/bin/../runtime" -llpython_runtime -lm' failed.

So, I pass the file (bindc_03b.c) and ran it by hand:

$ gcc -o bindc_03.out bindc_03__tmp__generated__.c  -I /home/thirumalai/Open_Source/lpython/src/bin/../libasr/runtime -L"/home/thirumalai/Open_Source/lpython/src/bin/../runtime" -Wl,-rpath,"/home/thirumalai/Open_Source/lpython/src/bin/../runtime" -llpython_runtime -lm integration_tests/bindc_03b.c && ./bindc_03.out

Is there an alternative for this?

czgdp1807 · 2023-04-18T13:25:50Z

integration_tests/CMakeLists.txt

@@ -219,8 +219,8 @@ RUN(NAME bindc_01            LABELS cpython llvm c)
 RUN(NAME bindc_02            LABELS cpython llvm c)
 RUN(NAME bindc_04            LABELS llvm c)
 RUN(NAME bindc_07            LABELS cpython llvm c)
-RUN(NAME exit_01             LABELS cpython llvm c wasm wasm_x86 wasm_x64)
-RUN(NAME exit_02     FAIL    LABELS cpython llvm c wasm wasm_x86 wasm_x64)


I think you should not de-register test for certain backends. That would be wrong. Instead create a new one and register it with limited backends.

czgdp1807 · 2023-04-18T13:28:18Z

tests/tests.toml

@@ -230,8 +230,8 @@ wat = true
 filename = "loop2.py"
 ast = true
 asr = true
-cpp = true
 c = true


Create loop3.py and register it with c = true, asr = true. Don't de-register a test from cpp.
Also do we really need ast = true for loop2.py as we are already testing it asr = true which will only work if AST is correct, no?

The same happens here as well: https://github.com/lcompilers/lpython/pull/1679/files#r1170193282

- Rename argv to _argv - Initialize the argv values in C backend using `_lpython_set_argv` function call

Thirumalai-Shaktivel · 2023-04-24T04:28:04Z

This PR is ready for review.

Thirumalai-Shaktivel · 2023-04-24T04:53:27Z

Thanks for the review!

Thirumalai-Shaktivel force-pushed the argv branch 3 times, most recently from d209fb0 to 58945ec Compare April 8, 2023 16:38

Thirumalai-Shaktivel force-pushed the argv branch from 58945ec to ae3c519 Compare April 9, 2023 06:14

Thirumalai-Shaktivel commented Apr 9, 2023

View reviewed changes

certik reviewed Apr 11, 2023

View reviewed changes

Thirumalai-Shaktivel force-pushed the argv branch from ccea854 to 38065b5 Compare April 13, 2023 04:39

Thirumalai-Shaktivel force-pushed the argv branch from 4aff31e to ebbf658 Compare April 18, 2023 10:33

Thirumalai-Shaktivel marked this pull request as ready for review April 18, 2023 10:33

czgdp1807 requested changes Apr 18, 2023

View reviewed changes

czgdp1807 marked this pull request as draft April 18, 2023 13:28

Thirumalai-Shaktivel force-pushed the argv branch 2 times, most recently from 832d6b5 to aa82f63 Compare April 23, 2023 11:37

Thirumalai-Shaktivel added 2 commits April 23, 2023 17:07

Use global list[str] type for argv,

e611337

- Rename argv to _argv - Initialize the argv values in C backend using `_lpython_set_argv` function call

Pass runtime_library path for the C backend to link executables

6beb6f1

Thirumalai-Shaktivel force-pushed the argv branch from aa82f63 to ccae820 Compare April 23, 2023 11:39

Set 0 to start and length if they are null in dims.

d690003

Thirumalai-Shaktivel force-pushed the argv branch 2 times, most recently from 5973835 to 7d970f9 Compare April 23, 2023 12:10

Add tests and update the refs.

710faf2

Thirumalai-Shaktivel force-pushed the argv branch from 7d970f9 to 710faf2 Compare April 23, 2023 12:22

Thirumalai-Shaktivel marked this pull request as ready for review April 23, 2023 13:13

czgdp1807 approved these changes Apr 24, 2023

View reviewed changes

Thirumalai-Shaktivel merged commit 3d3a53c into lcompilers:main Apr 24, 2023

Thirumalai-Shaktivel deleted the argv branch April 24, 2023 04:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assign the `_lpython_argv()` value to a global `list[str]` variable #1679

Assign the `_lpython_argv()` value to a global `list[str]` variable #1679

Thirumalai-Shaktivel commented Apr 7, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 7, 2023

Smit-create commented Apr 8, 2023

certik commented Apr 8, 2023

Thirumalai-Shaktivel commented Apr 9, 2023

Thirumalai-Shaktivel Apr 9, 2023

czgdp1807 Apr 18, 2023

Thirumalai-Shaktivel Apr 18, 2023

czgdp1807 Apr 19, 2023

Shaikh-Ubaid Apr 19, 2023

czgdp1807 Apr 19, 2023

Thirumalai-Shaktivel Apr 23, 2023

Thirumalai-Shaktivel commented Apr 10, 2023

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 10, 2023

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

certik Apr 11, 2023

certik commented Apr 11, 2023

certik commented Apr 12, 2023

certik commented Apr 12, 2023

Thirumalai-Shaktivel commented Apr 13, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 18, 2023 •

edited

Loading

czgdp1807 Apr 18, 2023

czgdp1807 Apr 18, 2023

Thirumalai-Shaktivel Apr 18, 2023

Thirumalai-Shaktivel Apr 23, 2023

Thirumalai-Shaktivel commented Apr 24, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 24, 2023

		@@ -7,8 +7,16 @@ def exit(error_code: i32):

		quit(error_code)

		# >>> argv >>>

Assign the _lpython_argv() value to a global list[str] variable #1679

Assign the _lpython_argv() value to a global list[str] variable #1679

Conversation

Thirumalai-Shaktivel commented Apr 7, 2023 • edited Loading

Thirumalai-Shaktivel commented Apr 7, 2023

Smit-create commented Apr 8, 2023

certik commented Apr 8, 2023

Thirumalai-Shaktivel commented Apr 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thirumalai-Shaktivel commented Apr 10, 2023

Thirumalai-Shaktivel commented Apr 10, 2023 • edited Loading

Thirumalai-Shaktivel commented Apr 10, 2023 • edited Loading

Thirumalai-Shaktivel commented Apr 10, 2023

Thirumalai-Shaktivel commented Apr 10, 2023 • edited Loading

Choose a reason for hiding this comment

certik commented Apr 11, 2023

certik commented Apr 12, 2023

certik commented Apr 12, 2023

Thirumalai-Shaktivel commented Apr 13, 2023 • edited Loading

Thirumalai-Shaktivel commented Apr 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thirumalai-Shaktivel commented Apr 24, 2023 • edited Loading

Thirumalai-Shaktivel commented Apr 24, 2023

Assign the `_lpython_argv()` value to a global `list[str]` variable #1679

Assign the `_lpython_argv()` value to a global `list[str]` variable #1679

Thirumalai-Shaktivel commented Apr 7, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 10, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 13, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 18, 2023 •

edited

Loading

Thirumalai-Shaktivel commented Apr 24, 2023 •

edited

Loading