[WIP] Various new features: Enhanced JIT Compiler targeted to multiple devices, ONNX Support, et al.

[hikettei]

Refactors

• Reduce code to less than 10000 lines.
• Remove cl-waffe2-simd, with keeping AVX/Neon/SLEEF intrinsic supports.
• Remove JITCPUTensor instead of adding backends/aten.

AbstractNodes

• Changes on base-impl
  • Unified and simplified arithmetic operations using !view.
    • cl-waffe2 uses broadcast+arithmetic ops instead of removing ScalarAdd/ScalarSub/ScalarMul/ScalarDiv/InverseTensorNode.
  • Rename: !inv -> !reciprocal
  • Fix typo: !leaky-relu
• keepdims option for !max/!min etc

Command Line Tool

• Uses Fiveam -> Rove as a testing tool.
• Make all tests hardware-independent
• Added a command line tool at ./roswell/waffe2.ros
• From ONNX to C Interpreter mode (waffe2 codegen xxx)
  • It generates the minimum C interpreter (calling CUDA/Metal functions), minimizing the dependencies.

ShapeTracker

• [Ehhancement] [ ] to express ScalarTensor, instead of using [scal] and out-scalar-p=t.

Frontend

• Implement from onnx to cl-waffe2 IR mode.

Backend

• Ready for implementing AbstractTensor backend

  • Fast Conv2d kernel (reduces memory-latency, winograd)
  • Ready for implementing GPU Supports, including Triton, Metal, CUDA, etc.

• Remove cl-waffe2-simd

• Remove obsolete SIMD dependencies

• Merge the aten-runtime branch.

APIs

• Implement AbstractListTensor

  • It basically behaves (Shape) Tensor.
  • but when called with forward, it wraps λ function w/ loop.
  • When called with !concatenate, it creates (LazyXXX ~ Shape) where LazyXXX = (length x) tensor
  • It enables implementing kv-cache for transformer models.

• Control-FLow

  • IfNode IfNode ~ EndIfNode
  • LoopNode LoopNode ~ EndLoopNode
  • POV: Is wf2IR turing-complete?

[hikettei]

I will merge the changes once as it is unlikely to get much development time for a while T_T. (plus, the IR specifications I have formulated are so bad that it is difficult to add new features. I need to take enough time and redevelop the entire back-end from scratch. but is it worth it?...)