Re-organise our building systems

[PsiACE]

We now rely on the makefile as part of the build system to assist developers.

• the model is relatively complex and we have to maintain cross-platform scripts, dependencies, and wrapped instructions.
• this part lacks explanatory documentation telling developers how and why to do so, and some of the commands are even redundant.

Goal

• reduce control over the developers' environment
• simplify workflow,only keep the necessary and important commands
• add documentation to fill in possible details
• align the continuous integration with the build model for local environment

What needs to be changed

• docker/
• scripts/
• tests/
• Makefile

[Xuanwo]

Is it a goal to remove all python scripts and rewrite them into a separate rust crate so that we don't need to depend on python?

[PsiACE]

This could be the result. 😄

[ZhiHanZ]

related to issue #2647
we could make our test framework similar to other cloud native libraries such as prometheus.
pros of adopting this approach:

1. Uniform operation to run integration tests, and remove the need for a lot of CI scripts and cluster setup scripts
2. Remove python dependencies and mysql dependencies( run query through bendctl query "")
3. For environment setup, most of operation is finished through rust toolchain setup, which should be cleaner.

[Xuanwo]

Here is a bad case of our build system: #3499

Take check https://github.com/datafuselabs/databend/runs/4545281599?check_suite_focus=true for a look.

First of all:

nohup target/${BIN}/databend-meta --single=true --log-level=ERROR &

Hide the error of databend-meta, so our check can't exit at the correct time.

Then, the complex deploy/test logic makes it harder to figure what happened.

[PsiACE]

We really need a clean and unified programme.

[BohuTANG]

Please do it!

[ZhiHanZ]

I think it is part of task of test framework

[Xuanwo]

Is there a proposal for our test framework?

[ZhiHanZ]

#2647 , it is in design stage, and definitely will have a proposal before 0.7 release

[Xuanwo]

Also related to #3518