Skip to content

Commit

Permalink
update readme (ingonyama-zk#303)
Browse files Browse the repository at this point in the history 
* refactor

* refactor

* refactor: add rust docs

* Update README.md

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>

* Update README.md

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>

* refactor

* Update icicle/README.md

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>

* Update wrappers/rust/README.md

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>

* Update README.md

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>

* refactor

* remove

---------

Co-authored-by: Jeremy Felder <jeremy.felder1@gmail.com>
  • Loading branch information
@ImmanuelSegol @jeremyfelder
ImmanuelSegol and jeremyfelder committed Dec 19, 2023
1 parent d5dd16d commit dcaa0b4
Show file tree
Hide file tree
Showing 4 changed files with 113 additions and 174 deletions.
171 changes: 25 additions & 146 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,151 +43,32 @@ ICICLE is a CUDA implementation of general functions widely used in ZKP. ICICLE

### Prerequisites

- [NVCC] (version 12.0 or newer)
- cmake 3.18 and above
- follow [these instructions](https://github.com/ingonyama-zk/icicle/tree/main/icicle#prerequisites-on-ubuntu)
- Any Nvidia GPU
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) version 12.0 or newer.
- [CMake]((https://cmake.org/files/)), version 3.18 and above. Latest version is recommended.
- [GCC](https://gcc.gnu.org/install/download.html) version 9, latest version is recommended.
- Any Nvidia GPU (which supports CUDA Toolkit version 12.0 or above).

If you don't have access to a Nvidia GPU check out [google-colab](#google-colab). If you require more compute power and are looking to build or do research with ICICLE refer to our [grant program][GRANT_PROGRAM].
> [!NOTE]
> It is possible to use CUDA 11 for cards which dont support CUDA 12, however we dont offically support this version and in the future there may be issues.
### Accessing Hardware

### Steps
If you don't have access to a Nvidia GPU we have some options for you.

1. Define or select a curve for your application; we've provided a [template][CRV_TEMPLATE] for defining a curve
2. Include the curve in [`curve_config.cuh`][CRV_CONFIG]
3. Now you can build the ICICLE library using nvcc
Checkout [Google Colab](https://colab.google/). Google Colab offers a free [T4 GPU](https://www.nvidia.com/en-us/data-center/tesla-t4/) instance and ICICLE can be used with it, reference this guide for setting up your [Google Colab workplace][GOOGLE-COLAB-ICICLE].

```sh
mkdir -p build
nvcc -o build/<binary_name> ./icicle/curves/index.cu -lib -arch=native
```

### Testing the CUDA code

We are using [googletest] library for testing. To build and run [the test suite](./icicle/README.md) for finite field and elliptic curve arithmetic, run from the `icicle` folder:

For testing, ensure the `BUILD_TESTS` option is enabled in cmake. If not, toggle it on by adding `-DBUILD_TESTS=ON` in the cmake configuration command:

```sh
cmake -S . -B build -DBUILD_TESTS=ON
```

Proceed with the following commands:

```sh
mkdir -p build
cmake -S . -B build
cmake --build build
cd build && ctest
```

NOTE: If you are using cmake versions < 3.24 add `-DCUDA_ARCH=<target_cumpute_arch>` to the command `cmake -S . -B build`


### Rust Bindings
If you require more compute and have an interesting research project, we have [bounty and grant programs][GRANT_PROGRAM].

For convenience, we also provide rust bindings to the ICICLE library for the following primitives:

- MSM
- NTT
- Forward NTT
- Inverse NTT
- ECNTT
- Forward ECNTT
- Inverse NTT
- Scalar Vector Multiplication
- Point Vector Multiplication
### Build systems

A custom [build script][B_SCRIPT] is used to compile and link the ICICLE library. The environment variable `ARCH_TYPE` is used to determine which GPU type the library should be compiled for and it defaults to `native` when it is not set allowing the compiler to detect the installed GPU type.

> NOTE: A GPU must be detectable and therefore installed if the `ARCH_TYPE` is not set.
Once you have your parameters set, run:

```sh
cargo build --release
```
ICICLE has three build systems.

You'll find a release ready library at `target/release/libicicle_utils.rlib`.
- [ICICLE core][ICICLE-CORE], C++ and CUDA
- [ICICLE Rust][ICICLE-RUST] bindings
- [ICICLE Golang][ICICLE-GO] bindings

To benchmark and test the functionality available in RUST, run:

```
cargo bench
cargo test -- --test-threads=1
```

The flag `--test-threads=1` is needed because currently some tests might interfere with one another inside the GPU.

### Example Usage

An example of using the Rust bindings library can be found in our [fast-danksharding implementation][FDI]

### Supporting Additional Curves

Supporting additional curves can be done as follows:

Create a JSON file with the curve parameters. The curve is defined by the following parameters:
- ``curve_name`` - e.g. ``bls12_381``.
- ``modulus_p`` - scalar field modulus (in decimal).
- ``bit_count_p`` - number of bits needed to represent `` modulus_p`` .
- ``limb_p`` - number of (32-bit) limbs needed to represent `` modulus_p`` (rounded up).
- ``ntt_size`` - log of the maximal size subgroup of the scalar field.
- ``modulus_q`` - base field modulus (in decimal).
- ``bit_count_q`` - number of bits needed to represent `` modulus_q`` .
- ``limb_q`` - number of (32-bit) limbs needed to represent `` modulus_q`` (rounded up).
- ``weierstrass_b`` - `b` of the curve in Weierstrauss form.
- ``weierstrass_b_g2_re`` - real part of the `b` value in of the g2 curve in Weierstrass form.
- ``weierstrass_b_g2_im`` - imaginary part of the `b` value in of the g2 curve in Weierstrass form.
- ``gen_x`` - `x` coordinate of a generator element for the curve.
- ``gen_y`` - `y` coordinate of a generator element for the curve.
- ``gen_x_re`` - real part of the `x` coordinate of generator element for the g2 curve.
- ``gen_x_im`` - imaginary part of the `x` coordinate of generator element for the g2 curve.
- ``gen_y_re`` - real part of the `y` coordinate of generator element for the g2 curve.
- ``gen_y_im`` - imaginary part of the `y` coordinate of generator element for the g2 curve.
- ``nonresidue`` - nonresidue, or `i^2`, or `u^2` - square of the element that generates quadratic extension field of the base field.

Here's an example for BLS12-381.
```
{
"curve_name" : "bls12_381",
"modulus_p" : 52435875175126190479447740508185965837690552500527637822603658699938581184513,
"bit_count_p" : 255,
"limb_p" : 8,
"ntt_size" : 32,
"modulus_q" : 4002409555221667393417789825735904156556882819939007885332058136124031650490837864442687629129015664037894272559787,
"bit_count_q" : 381,
"limb_q" : 12,
"weierstrass_b" : 4,
"weierstrass_b_g2_re" : 4,
"weierstrass_b_g2_im" : 4,
"gen_x" : 3685416753713387016781088315183077757961620795782546409894578378688607592378376318836054947676345821548104185464507,
"gen_y" : 1339506544944476473020471379941921221584933875938349620426543736416511423956333506472724655353366534992391756441569,
"gen_x_re" : 352701069587466618187139116011060144890029952792775240219908644239793785735715026873347600343865175952761926303160,
"gen_x_im" : 3059144344244213709971259814753781636986470325476647558659373206291635324768958432433509563104347017837885763365758,
"gen_y_re" : 1985150602287291935568054521177171638300868978215655730859378665066344726373823718423869104263333984641494340347905,
"gen_y_im" : 927553665492332455747201965776037880757740193453592970025027978793976877002675564980949289727957565575433344219582,
"nonresidue" : -1
}
```

Save the parameters JSON file under the ``curve_parameters`` directory.

Then run the Python script ``new_curve_script.py `` from the root folder:

```
python3 ./curve_parameters/new_curve_script.py ./curve_parameters/bls12_381.json
```

The script does the following:
- Creates a folder in ``icicle/curves`` with the curve name, which contains all of the files needed for the supported operations in cuda.
- Adds the curve's exported operations to ``icicle/curves/index.cu``.
- Creates a file with the curve name in ``src/curves`` with the relevant objects for the curve.
- Creates a test file with the curve name in ``src``.

Also files from ``./icicle/curves/<curve_name>/supported_operations.cu`` should be added individually to ``add_library`` section of [``./icicle/CMakeLists.txt``][CMAKELISTS]

Testing the new curve could be done by running the tests in ``tests_curve_name`` (e.g. ``tests_bls12_381``).
ICICLE core always needs to be built as part of the other build systems as it contains the core ICICLE primitives implemented in CUDA. Reference these guides for the different build systems, [ICICLE core guide][ICICLE-CORE-README], [ICICLE Rust guide][ICICLE-RUST-README] and [ICICLE Golang guide][ICICLE-GO-README].

## Docker

Expand All @@ -198,14 +79,6 @@ docker build -t <name_of_your_choice> .
docker run --gpus all -it <name_of_your_choice> /bin/bash
```

## Google Colab

[Colab](https://colab.google/) is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources including GPUS!

You can easily run ICICLE in Google Colab on a free GPU instance, this is a great option for those who want to get started with ICICLE instantly without any local setup or GPU.

Follow this [guide][GOOGLE_COLAB_ICICLE] for more details.

## Contributions

Join our [Discord Server][DISCORD] and find us on the icicle channel. We will be happy to work together to support your use case and talk features, bugs and design.
Expand Down Expand Up @@ -252,14 +125,20 @@ See [LICENSE-MIT][LMIT] for details.
[CRV_TEMPLATE]: ./icicle/curves/curve_template/
[CRV_CONFIG]: ./icicle/curves/index.cu
[B_SCRIPT]: ./build.rs
[FDI]: https://github.com/ingonyama-zk/fast-danksharding
[LMIT]: ./LICENSE
[DISCORD]: https://discord.gg/Y4SkbDf2Ff
[googletest]: https://github.com/google/googletest/
[HOOKS_DOCS]: https://git-scm.com/docs/githooks
[HOOKS_PATH]: ./scripts/hooks/
[CMAKELISTS]: https://github.com/ingonyama-zk/icicle/blob/f0e6b465611227b858ec4590f4de5432e892748d/icicle/CMakeLists.txt#L28
[GOOGLE_COLAB_ICICLE]: https://github.com/gkigiermo/rust-cuda-colab
[GRANT_PROGRAM]: https://docs.google.com/forms/d/e/1FAIpQLSc967TnNwxZZ4akejcSi4KOUmGrEc68ZZV-FHLfo8KnP1wbpg/viewform
[GOOGLE-COLAB-ICICLE]: https://github.com/gkigiermo/rust-cuda-colab
[GRANT_PROGRAM]: https://medium.com/@ingonyama/icicle-for-researchers-grants-challenges-9be1f040998e
[ICICLE-CORE]: ./icicle/
[ICICLE-RUST]: ./wrappers/rust/
[ICICLE-GO]: ./goicicle/
[ICICLE-CORE-README]: ./icicle/README.md
[ICICLE-RUST-README]: ./wrappers/rust/README.md
[ICICLE-GO-README]: ./goicicle/README.md


<!-- End Links -->
53 changes: 34 additions & 19 deletions goicicle/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,27 +1,46 @@
# ICICLE CUDA to Golang Binding Guide
# Golang Bindings

This guide provides instructions on how to compile CUDA code using the provided Makefile, and then how to use the resulting shared libraries to bind Golang to ICICLE's CUDA code.
To build the shared library:

## Prerequisites
To build shared libraries for all supported curves.

To compile the CUDA files, you will need:
```
make all
```

If you wish to build for a specific curve, for example bn254.

```
make libbn254.so
```

The current supported options are `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so`. The resulting `.so` files are the compiled shared libraries for each curve.

Finally to allow your system to find the shared libraries

```
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH/<path_to_shared_libs>
```

- CUDA toolkit installed. The Makefile assumes CUDA is installed in `/usr/local/cuda`. If CUDA is installed in a different location, please adjust the `CUDA_ROOT_DIR` variable accordingly.
- A compatible GPU and corresponding driver installed on your machine.
## Running golang tests

## Structure of the Makefile
To run the tests for curve bn254.

The Makefile is designed to compile CUDA files for four curves: BN254, BLS12_381, BLS12_377 and BW6_671. The source files are located in the `icicle/curves/` directory.
```
go test ./goicicle/curves/bn254 -count=1
```

## Cleaning up

## Compiling CUDA Code
If you want to remove the compiled files

1. Navigate to the directory containing the Makefile in your terminal.
2. To compile all curve libraries, use the `make all` command. This will create four shared libraries: `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so`.
3. If you want to compile a specific curve, you can do so by specifying the target. For example, to compile only the BN254 curve, use `make libbn254.so`. Replace `libbn254.so` with `libbls12_381.so`, `libbls12_377.so` or `libbw6_671.so` to compile those curves instead.
```
make clean
```

The resulting `.so` files are the compiled shared libraries for each curve.
This will remove all shared libraries generated from the `make` file.

## Golang Binding
# How do Golang bindings work?

The shared libraries produced from the CUDA code compilation are used to bind Golang to ICICLE's CUDA code.

Expand All @@ -44,11 +63,7 @@ func main() {

Replace `/path/to/shared/libs` with the actual path where the shared libraries are located on your system.

## Cleaning up

If you want to remove the compiled files, you can use the `make clean` command. This will remove the `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so` files.

## Common issues
# Common issues

### Cannot find shared library

Expand Down
26 changes: 17 additions & 9 deletions icicle/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,14 @@
## Running tests

```sh
mkdir -p build; cmake -S . -B build; cmake --build build; cd build && ctest; cd ..
mkdir -p build;
cmake -DBUILD_TESTS=ON -DCURVE=<support_curve> -S . -B build;
cmake --build build;
cd build && ctest;
cd ..
```

## Prerequisites on Ubuntu

Before proceeding, make sure the following software installed:

1. CMake at least version 3.18, which can be downloaded from [cmake.org](https://cmake.org/files/)
It is recommended to have the latest version installed.
2. [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu) version 12.0 or newer.
3. GCC - version 9 or newer recommended.
The command above will build ICICLE Core and run the ctest.

## Troubleshooting

Expand Down Expand Up @@ -81,3 +78,14 @@ Otherwise create a symlink. For example, if the CUDA toolkit is installed with a
Alternatively, you can replace the old CUDA root with a symlink to the new CUDA installation with the following command:
`ln -sf /usr/local/cuda-12.1/ /usr/lib/nvidia-cuda-toolkit/`
### 8 - Error while loading shared libraries
`cmake: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory`
Make sure `libssl` is installed.
```
sudo apt-get update
sudo apt-get install libssl1.0.0 libssl-dev
```
37 changes: 37 additions & 0 deletions wrappers/rust/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
### Rust Bindings

`icicle-core` defines all interfaces, macros and common methods.

`icicle-cuda-runtime` defines `DeviceContext` which can be used to manage a specific GPU as well as wrapping common CUDA methods.

`icicle-curves` implements all interfaces and macros from `icicle-core` for each curve. For example `icicle-bn254` implements curve `bn254`. Each curve has its own [build script](./icicle-curves/icicle-bn254/build.rs) which will build the CUDA libraries for that curve as part of the rust-toolchain build.

## Building a curve and running tests

Enter a curve implementation.

```
cd icicle-curves/icicle-bn254
```

To build

```sh
cargo build --release
```

The build may take a while because we are also building the CUDA libraries for the selected curve.

To run benchmarks

```
cargo bench
```

To run test

```sh
cargo test -- --test-threads=1
```

The flag `--test-threads=1` is needed because currently some tests might interfere with one another inside the GPU.

0 comments on commit dcaa0b4

Please sign in to comment.