Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add flink source and sink ci test #252

Merged
merged 6 commits into from
Jun 9, 2023
Merged

add flink source and sink ci test #252

merged 6 commits into from
Jun 9, 2023

Conversation

F-PHantam
Copy link
Contributor

@F-PHantam F-PHantam commented Jun 9, 2023

main work in this pr :

  1. On the basis of the original MySQL CDC, a basic table has been added, which is mostly full data type (except for JSON, time, and YAER that require special processing);
  2. Flink source reads the LakeSoul cdc table and then writes it to the LakeSoul table. Then, the data written to the LakeSoul table by Flink is compared with MySQL data. The data in MySQL table will be added, deleted, and updated, and data comparison is performed every time;
  3. Flink DataGen generates some data on its own and writes it to CSV and LakeSoul table, without primary keys. Then Spark reads the CSV data and compares the LakeSoul table data.

@F-PHantam F-PHantam requested review from moresun, xuchen-plus and Ceng23333 and removed request for moresun June 9, 2023 08:21
@F-PHantam F-PHantam added the flink flink support into lakesoul label Jun 9, 2023
@xuchen-plus xuchen-plus changed the title add flink source and sink cli add flink source and sink ci test Jun 9, 2023
@xuchen-plus xuchen-plus merged commit 9e196f7 into lakesoul-io:main Jun 9, 2023
F-PHantam added a commit to F-PHantam/LakeSoul that referenced this pull request Jul 3, 2023
Signed-off-by: fphantam <dongf@dmetasoul.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flink flink support into lakesoul
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

2 participants