Add ability to configure static buffer capacity

[dartdart26]

Currently, StaticBuffer in env/src/engine/on_chain/buffer.sh has a
static (constant) size of 16 kB. However, there are cases which might
require more space. One such case is passing in and getting out bigger
buffers to/from a chain extension. As chain extensions can be arbitrary,
it makes sense to have the ability to change the size of the buffer.

This commit adds the ability to configure the static buffer capacity at
compile time through the ENV_ENGINE_STATIC_BUFFER_CAPACITY environment
variable. For example, to double it to 32 kB, one might do the following
when compiling a smart contract:

ENV_ENGINE_STATIC_BUFFER_CAPACITY=32768 cargo +nightly contract build

In order to implement, we introduce the ink_build_script_utils crate
that has functions to read the value from the environment variable and
convert it to a Rust constant in a generated file at build-time. Then,
we utilize a build.rs script at the env crate level.

Documentation about the ENV_ENGINE_STATIC_BUFFER_CAPACITY environment
variable can be added in a separate commit to the ink-docs repo.

[HCastano]

Before allowing this we'll need to think more about the performance impacts of increasing the buffer size. I think there are places where we encode/decode the entire buffer for instance.

Have you run into this issue yet? If so, can you share the code?

[dartdart26]

Thank you for pointing that out @HCastano. I haven't run into the issue, but I haven't done extensive testing and benchmarking.

The use case is implementing computation on encrypted data on-chain using fully homomorphic encryption (FHE). The issue is that ciphertexts are bigger as compared to the corresponding plaintexts. Therefore, when doing operations on state and inputs that are ciphertexts via a chain extension, bigger buffers are needed. For example, smart contract code might want to accept a ciphertext as an input, send it to a chain extension, get the resulting ciphertext and store it in storage.

[ascjones]

Did you already consider adding this as a const to the Environment trait?

I realize that adding that E: Environment trait bound to StaticBuffer could be hairy but would be good to rule it out before adding build script hacks.

[dartdart26]

Did you already consider adding this as a const to the Environment trait?

I realize that adding that E: Environment trait bound to StaticBuffer could be hairy but would be good to rule it out before adding build script hacks.

That would be pretty nice, actually. You just create a custom environment and specify it there, as part of code. I don't currently know what it takes to do it, but will try it. Thanks!

[xgreenx]

Did you already consider adding this as a const to the Environment trait?

I realize that adding that E: Environment trait bound to StaticBuffer could be hairy but would be good to rule it out before adding build script hacks.

It will require propagation Environment everywhere. Right now, it is only needed for methods in TypedEnvBackend. Adding it into EnvBackend means we need to propagate it throw all push/pull logic.

[xgreenx]

I created #1303 with a description of the idea of how to make all things from Environment more user-friendly. The buffer capacity also can be part of ink_types.

[dartdart26]

@ascjones I looked at different parts of code and, essentially, I am not exactly sure how to get the actual Environment type in the ink_env crate to use in the StaticBuffer declaration. I would appreciate any hints.

For example, I looked at lang::reflect and the way it gets it, but I think using it requires substantial refactoring, because ink_lang depends on ink_env. Another option is to maybe have a macro in ink_env that generates the StaticBuffer type for every contract. Do you think that's actually an option?

I also looked at somehow taking StaticBuffer out of ink_env, but I think that is equivalent to getting the Environment type as in lang::reflect.

As @xgreenx mentioned, on_chain::EnvInstance contains the buffer, but the environment is currently used for TypedEnvBackend methods. Again, not completely sure how to transform it such that we can take the constant for the StaticBuffer declaration.

I realize I may be missing something and it might turn out much simpler.

[cmichi]

@dartdart26 Sorry for dropping the ball on this. Could you take a look at the suggestion in #1471? Would you be up for implementing it like this?

[SkymanOne]

@ascjones I looked at different parts of code and, essentially, I am not exactly sure how to get the actual Environment type in the ink_env crate to use in the StaticBuffer declaration. I would appreciate any hints.

For example, I looked at lang::reflect and the way it gets it, but I think using it requires substantial refactoring, because ink_lang depends on ink_env. Another option is to maybe have a macro in ink_env that generates the StaticBuffer type for every contract. Do you think that's actually an option?

I also looked at somehow taking StaticBuffer out of ink_env, but I think that is equivalent to getting the Environment type as in lang::reflect.

As @xgreenx mentioned, on_chain::EnvInstance contains the buffer, but the environment is currently used for TypedEnvBackend methods. Again, not completely sure how to transform it such that we can take the constant for the StaticBuffer declaration.

I realize I may be missing something and it might turn out much simpler.

Hey. Would like some help with your PR? Did you have a chance to look at Michi's suggestion?

[cmichi]

@dartdart26 I'm closing this PR as it seems to have staled, please see the comments above.

@SkymanOne Can you please assign yourself #1471 and implement it this way? I think it should mostly be about adding the trait bound in a lot of places.