Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add eth_simulateV1 #484

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open

Conversation

KillariDev
Copy link

@KillariDev KillariDev commented Nov 4, 2023

eth_simulateV1

This PR is a cleaned up version from #383. The #383 is based heavily on #312 from @s1na (eth_batchCall). This spec encompasses some of the ideas presented by @MicahZoltu @OlegJakushkin, @LukaszRozmej, @KillariDev, @epheph and others.

Summary

This introduces a new RPC method eth_multicallV1 allows users to make complex RPC calls to Ethereum nodes. Compared to eth_call, eth_multicallV1 has following extra features:

  • You can encapsulate multiple dependent calls in a single call
  • The calls happen inside blocks. You can simulate multiple blocks that can be arbitrary far from each other
  • Block variables can be overridden (e.g. time)
  • Account state can be overridden for every block (e.g. code and balance)
  • It is possible to override precompiles (e.g. ecrecover) with arbitrary EVM code
  • ETH transfers produce logs similar to ERC20 logs
  • Validation mode. You can choose to do very strict simulation or more relaxed one similar to eth_call

Motivation

The features of eth_multicallV1 allow many interesting simulation use cases such as:

  • To simulate simple erc20 approval & swap feature in a wallet, we need to be able to send multiple transactions
  • Searchers need to trigger oracle update and wait a number of blocks
  • Researchers need to trigger a price change and wait a number of blocks for a TWAP adjustment
  • Wallets are beginning to provide advanced simulation to the user
  • Block builders are beginning to explore multi-block MEV
  • Fake Permit2 or other EIP-712 approvals

This blog post also explains motivation around eth_simulate: https://mirror.xyz/killaridev.eth/NXR9v4r8b4SWl-hbqJqRQyIAcE4GACobzMObREvL0fo and a PEEPanEIP episode: https://www.youtube.com/watch?v=4uZyQQ6qz4U

Implementation

Geth: https://github.com/s1na/go-ethereum/tree/multicall
Nethermind: https://github.com/NethermindEth/nethermind

@KillariDev
Copy link
Author

Here's slides for multicall presentation: https://docs.google.com/presentation/d/1lEaqHTY3ud8pe6VAFwLkb-jdoHpTMBfuF1K9OKy-azs

Copy link
Member

@jochem-brouwer jochem-brouwer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got some comments :) I love this idea, this will make UX so much better!

docs/multicall-notes.md Outdated Show resolved Hide resolved
docs/multicall-notes.md Outdated Show resolved Hide resolved
| input | no data |
| gasPrice | `0x0` |
| maxPriorityFeePerGas | `0x0` |
| maxFeePerGas | `0x0` |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These default values will crash, since baseFeePerGas is always 7 or higher. These default values will be rejected by blockchain spec, since tx is not willing to pay base fee. Note that the default block baseFeePerGas is calculated according to spec, so this will be 7 or higher always.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: change maxFeePerGas and gasPrice (in case of tx type 0/1) to baseFeePerGas of the block and keep maxPriorityFeePerGas 0.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These default values will crash, since baseFeePerGas is always 7 or higher. These default values will be rejected by blockchain spec, since tx is not willing to pay base fee. Note that the default block baseFeePerGas is calculated according to spec, so this will be 7 or higher always.

yeah, this will crash on validation mode. On non-validation mode, we will allow you to make transactions that have zero basefee (similar to eth_call)

| accessList | empty array |

## Overriding default values
The default values of blocks and transactions can be overriden. For Transactions we allow overriding of variables `type`, `nonce`, `to`, `from`, `gas limit`, `value`, `input`, `gasPrice`, `maxPriorityFeePerGas`, `maxFeePerGas`, `accessList`, and for blocks we allow modifications of `number`, `time`, `gasLimit`, `feeRecipient`, `prevRandao` and `baseFeePerGas`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slight nit: for completeness, withdrawals should also be able to be overridden (to simulate withdrawals).

docs/multicall-notes.md Outdated Show resolved Hide resolved
ETH logs are not part of the calculation for logs bloom filter. Also, similar to normal logs, if the transaction sends ETH but the execution reverts, no log gets issued.

## Validation
The multicall has a feature to enable or disable validation with setting `Validation`, by default, the validation is off, and the multicall mimics `eth_call` with reduced number of checks. Validation enabled mode is intended to give as close as possible simulation of real EVM block creation, except there's no checks for transaction signatures and we also allow one to send a direct transaction from a contract.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not entirely sure what this would remove/add. If I turn this on, in what situation would this be useful, and what would change if I would have kept it off? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For instance, if validation is set to off, are tx gas limits not checked? (So I can include a tx with gas limit higher than the block gas limit?).

Copy link
Member

@jochem-brouwer jochem-brouwer Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, it is in execute.yaml

    validation:
      title: Validation
      description: |-
        When true, the multicall does all validations that a normal EVM would do, except contract sender and signature checks. When false, multicall behaves like eth_call.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is bit ill defined (as discussed in todays call). Because Nethermind and Geth behave bit differently with eth_call and we have been trying to figure out the differences so we could document them. The intention that is that when validation is enabled, the clients would behave exactly the same, while not fixing the issue that with non-validation, the validation rules might differ a bit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I think this spec should explicitly state what should be not be validated if the validation is off. For instance, currently it only seems that you can override the sender (so you can create unsigned txs). But, it also seems that it is possible to set base fee to zero (so do not validate this as well). It should be clearly defined what should be validated and what not. (And this is non-trivial since there is a huge ruleset of what should be validated in transactions/blocks)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For validation mode, I'm fine with base fee not being allowed to go below 7.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it makes sense that much to validate base fee in validation mode if the user specifically sets the base fee to something. Otherwise we could remove all overrides for validation mode, as none of them can happen on mainnet.

Copy link
Contributor

@s1na s1na Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly also create multiple validation modes / flags if necessary?

Please no! In fact I want to argue that we should remove validation flag and default to relax-mode. My argument is that this is easy to add in later since we have designed the API param to be an object. It will be a backwards-compatible change. I'd personally like to hear from some people to say they need this first.

That said, we should still define exactly what is not validated in relax-mode. So I think that convo is important to have. And I will share here exactly what geth is validating:

For block validation, only these fields are validated:

  • Only block number and timestamp are validated to be incremental.

As for tx validation, these are NOT validated:

  • Signature is not validated
  • Nonce too high, too low, out of bounds are not validated
  • Sender is EoA is not validated
  • If gas price fields are 0, then sender doesn't need to have enough balance for gas (only value)
  • If blobGasFeeCap is 0, then blob gas requirement is skipped

Copy link
Contributor

@s1na s1na Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add that these relaxations are all cause for a transaction not to go through. I.e. if I have a call that works in the relax-mode, sign it and submit it to a node, it will not be gossiped to the network at all.

I'd argue if there is any danger to a user it's from all of the flexibility we're providing through state and precompile overrides. E.g. they assume in the simulation to be in a certain condition by using overrides, and submit the transaction when that is not met. That tx will be mined and will revert (most probably).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd personally like to hear from some people to say they need this first.

Our team would like it because it allows us to verify that the transaction is properly constructed and will execute against mainnet without error. In some cases we ship a set of transactions as a "bundle" to searchers, and they need to all execute properly in sequence (including things like balance checks), and in other cases we ship a single transaction to a signer and we want a high degree of confidence that it will execute on-chain just like it did during simulation.

KillariDev and others added 4 commits November 27, 2023 16:11
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
Co-authored-by: Jochem Brouwer <jochembrouwer96@gmail.com>
An interesting note here is that an user can specify block numbers and times of some blocks, but not for others. When block numbers of times are left unspecified, the default values will be used. After the blocks have been constructed, and default values are calculated, the blocks are checked that their block numbers and times are still valid.

## ETH transfer logs
When `traceTransfers` setting is enabled on `eth_multicallV1` The multical will return logs for ethereum transfers along with the normal logs sent by contracts. The ETH transfers are identical to ERC20 transfers, except the "sending contract" is address `0x0`.
Copy link

@mds1 mds1 Dec 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest using 0xEeeeeEeeeEeEeeEeEeEeeEEEeeeeEeeeeeeeEEeE as the sending contract, since that's become a de facto approach for representing ETH as a token address within contracts.

Here's a simple sourcegraph search showing usage: https://sourcegraph.com/search?q=0xEeeeeEeeeEeEeeEeEeEeeEEEeeeeEeeeeeeeEEeE&groupBy=repo

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect if you were to do a similar search of 0x000... you would get even more results. "A lot of people use X" is quite different from "Most people use X".

If we ignore any past precedence, I feel like 0x000... makes it a tad more explicit that this is special, and not some precompile or something. For parameters, 0xEee... makes a bit of sense because 0 is the default value for a lot of stuff (e.g., null, missing, etc.) and thus often passed on accident, so using nonzero helps protect against a class of bugs. In the case of event sourcing, 0x000... isn't provided by user/developer so they cannot get it wrong.

That being said, I don't have a strong argument against 0xEeee..., only that I think 0x000... is marginally better and I suspect more widely used.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree a search for the zero address would yield more results, but only because the zero address is used for a much wider range of cases, whereas 0xEee... seems to only be used to represent ETH. This is anecdotal and it's hard to know concretely, but in my experience 0xEee... is more common for representing ETH than the zero address.

Personally, if I didn't know about the special ETH logs, seeing 0xEee... would make the log's meaning more intuitive and apparent than seeing 0x000....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree a search for the zero address would yield more results, but only because the zero address is used for a much wider range of cases

Sorry, I meant if you could somehow limit the search of 0x0000... to only places where it was being used as a representation of ETH I suspect it would be more/bigger. That being said, perhaps the searchability of 0xEeee... is a significant selling point.

Personally, if I didn't know about the special ETH logs, seeing 0xEee... would make the log's meaning more intuitive and apparent than seeing 0x000....

For what it is worth, this feature is disabled by default and you need to set a flag to turn it on. I'm not sure that is a meaningful argument though.

Copy link

@mmsaki mmsaki Dec 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree a search for the zero address would yield more results, but only because the zero address is used for a much wider range of cases, whereas 0xEee... seems to only be used to represent ETH. This is anecdotal and it's hard to know concretely, but in my experience 0xEee... is more common for representing ETH than the zero address.

I agree with @mds1 that 0xEee.. seems more appropriate for Eth transfer logs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the spec has since been updated to use 0xEee.....

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The notes still mentions 0x0 here: https://github.com/ethereum/execution-apis/pull/484/files#diff-1e4727bbfb75c5cc85054abbf972a1cd472e9e512a9d8bbc8152210a0062182bR116

Aside: It still feels "unclean" to me using a specific non-reserved address, but in researching this I was surprised to learn that EIP-1352 was never officially adopted.

Copy link

@sambacha sambacha Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0xEeeeeEeeeEeEeeEeEeEeeEEEeeeeEeeeeeeeEEeE is not correct if you're using EIP-1191, the correct chainId encoded address would be 0xeeeEEEeEEeeeEeEeEEEeEeeeeEEEEeEEeEeeeeeE

having it as 0x00... would be agnostic to chainId encoding.

Edit: this is a purely aesthetic choice, though I would find it helpful

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I and others have argued against EIP-1191 in the discussion for it: ethereum/EIPs#1121

My position is still largely the same. It is backward incompatible with EIP-55 checksumming, so I generally don't recommend apps checksum to 1191 until long after almost all tooling supports reading 1191 checksums.

Copy link

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This endpoint sounds very useful. I believe this is very similar to eth_callMany that some clients support: erigontech/erigon#4471

imo it makes sense to introduce a standardized endpoint.
If I understood this currently, then the calls in the batch are treated as separate transactions.
I think multicall could potentially be confusing because "mutlicall" is already widely used as

Multicall allows multiple smart contract constant function calls to be grouped into a single call and the results aggregated into a single result.

https://github.com/joshstevens19/ethereum-multicall#readme

@MicahZoltu
Copy link
Contributor

You are correct, this allows you to run multiple calls against one or more blocks in sequence (all building up the same state). Maybe eth_simulateTransactions or eth_simulateBlocks would be better?

@mds1
Copy link

mds1 commented Dec 19, 2023

I'm mostly indifferent to the name, but eth_batchCall or eth_simulateCalls are other alternatives that don't overload the multicall usage.

On this topic, why is the method proposed as eth_multicallV1 instead of eth_multicall? Most RPC methods do not have versioning in their name so this seems to deviate from convention. eth_signTypedData_v4 is the only one I know of with versioning

@MicahZoltu
Copy link
Contributor

On this topic, why is the method proposed as eth_multicallV1 instead of eth_multicall? Most RPC methods do not have versioning in their name so this seems to deviate from convention. eth_signTypedData_v4 is the only one I know of with versioning

The lack of versioning in existing JSON-RPC methods has been a thorn in the side of people wanting to make changes to them since essentially the beginning of Ethereum. Rather than continue to cause our future selves suffering to maintain a naming convention, we felt it would be wise to break the cycle and introduce versioning.

As for the naming convention, it follows the Engine API's naming convention (the communication channel between execution and consensus clients).

@mattsse
Copy link

mattsse commented Dec 20, 2023

but eth_batchCall or eth_simulateCalls

these are great alternative suggestions, would lean towards eth_simulateCalls, because batchCalls could be mistaken as batched eth_call

Using versioning for this sgtm!

@KillariDev
Copy link
Author

but eth_batchCall or eth_simulateCalls

these are great alternative suggestions, would lean towards eth_simulateCalls, because batchCalls could be mistaken as batched eth_call

Using versioning for this sgtm!

I think eth_simulateTransactionsV1 would the best. As you can simulate more than just calls, you can simulate transactions

src/schemas/execute.yaml Outdated Show resolved Hide resolved
@KillariDev KillariDev changed the title add eth_multicallV1 add eth_simulateV1 Jan 11, 2024
Copy link
Member

@jochem-brouwer jochem-brouwer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some new comments :) These might have already been discussed, sorry for that, then just dismiss/resolve them :)

Our solution to this problem is to define block hash of a phantom block to be:

```
keccak(rlp([hash_of_previous_non_phantom_block, phantom_block_number]))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these numbers left-padded? I think adding this would be clear (there are a lot of EIPs where the spec says to either left pad them or not)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assumed that hash_of_previous_non_phantom_block is a 256-bit integer value (meaning 32 bytes of data), but perhaps I'm wrong here? If so, that would mean it is "left padded" I guess, though that seems like a weird way to think about these values to me. Either way, clarification would be an improvement so I agree that we should provide some clarity here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I actually assumed it was not left-padded, so this should definitely be clarified :)

| sha3Uncles | Empty trie root |
| withdrawals | Empty array |
| uncles | Empty array |
| blobBaseFee | Calculated on what it should be according to EIP-4844 spec. Note: blobBaseFee is not adjusted in the phantom blocks. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a block property, we have excessBlobGas and blobGasUsed, but for this simulation I can understand why one would directly input the output of whatever calculates the blob gas fee (it is something that looks like an exponential with two inputs, and you dont want to "find" the right inputs I guess)

| baseFeePerGas | Calculated on what it should be according to Ethereum's spec. Note: baseFeePerGas is not adjusted in the phantom blocks. |
| sha3Uncles | Empty trie root |
| withdrawals | Empty array |
| uncles | Empty array |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: do we throw here on post-merge blocks if the uncles are non-empty? There can be no uncles post-merge and the uncle hash is therefore also stubbed to the empty trie hash.

| uncles | Empty array |
| blobBaseFee | Calculated on what it should be according to EIP-4844 spec. Note: blobBaseFee is not adjusted in the phantom blocks. |
| number | Previous block number + 1 |
| logsBloom | Calculated normally. ETH logs are not part of the calculation |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are ETH logs? For clarity it would be nice to link that these are explain further down 😄

| logsBloom | Calculated normally. ETH logs are not part of the calculation |
| receiptsRoot | Calculated normally |
| transactionsRoot | Calculated normally |
| size | Calculated normally |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size is not part of the block header (so I also don't see why it is necessary here?)

| nonce | Take the correct nonce for the account prior multicall and increment by one for each transaction by the account |
| to | `null` |
| from | `0x0000000000000000000000000000000000000000` |
| gas limit | (blockGasLimit - SumOfGasLimitOfTransactionsWithDefinedGasLimit) / NumberOfTransactionsWithoutKnownGasLimit |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this is smart! I had to play with this for a while to see if I could find an exception but obviously not, if you ensure the sum of all txs equals the block gas limit it works (ensure this is rounded down to really make sure there is no exception, so I would use // instead of / to property note this).

However there is an edge case: we can get a negative gas limit, and we can also get a gas limit which cannot cover the base gas costs of 21000 (or, if you include the calldata costs or the contract creation costs this is more). This edge case can lead to invalid blocks. (Since the txs are invalid)

The other case is the negative one, lets say we have 30M gas limit and we have 3 txs, one has gas limit 30M, the other has gas limit 30M as well (if we execute both they only use 100k gas), but then the 3rd one now has a negative gas limit of 30M which is obviously not possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I see we throw if this happens.

| value | `0x0` |
| input | no data |
| gasPrice | `0x0` |
| maxPriorityFeePerGas | `0x0` |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we throw for the fields set if these are not available for the tx type? (The 4 types below here, so maxPriorityFeePerGas, maxFeePerGas, accessList, blobVersionedHashes)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(they can be set and it is fine, but might be confusing for the end user)

| maxPriorityFeePerGas | `0x0` |
| maxFeePerGas | `0x0` |
| accessList | empty array |
| blobVersionedHashes | empty array |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blob txs cannot be simulated it seems per the current spec since it cannot be overridden.

| blobVersionedHashes | empty array |

## Overriding default values
The default values of blocks and transactions can be overriden. For Transactions we allow overriding of variables `type`, `nonce`, `to`, `from`, `gas limit`, `value`, `input`, `gasPrice`, `maxPriorityFeePerGas`, `maxFeePerGas`, `accessList`, and for blocks we allow modifications of `number`, `time`, `gasLimit`, `feeRecipient`, `prevRandao`, `baseFeePerGas` and `blobBaseFee`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if I have a specific use case that I want a block hash of a previous block be of some kind? Then I should be able to override hash as well.

What about withdrawals?


## Clients can set their own limits
Clients may introduce their own limits to prevent DOS attacks using the method. We have thought of three such standard limits
- How many blocks can be defined in `BlockStateCalls`. The suggested default for this is 256 blocks
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also argue that there is a max number, because if one allows to create only one block, then one can still request block very high number and this would impose a lot of phantom block hashings.

holiman added a commit to ethereum/go-ethereum that referenced this pull request Sep 6, 2024
This is a successor PR to #25743. This PR is based on a new iteration of
the spec: ethereum/execution-apis#484.

`eth_multicall` takes in a list of blocks, each optionally overriding
fields like number, timestamp, etc. of a base block. Each block can
include calls. At each block users can override the state. There are
extra features, such as:

- Include ether transfers as part of the logs
- Overriding precompile codes with evm bytecode
- Redirecting accounts to another address

## Breaking changes

This PR includes the following breaking changes:

- Block override fields of eth_call and debug_traceCall have had the
following fields renamed
  - `coinbase` -> `feeRecipient`
  - `random` -> `prevRandao`
  - `baseFee` -> `baseFeePerGas`

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
mattsse pushed a commit to paradigmxyz/revm-inspectors that referenced this pull request Sep 12, 2024
ref ethereum/execution-apis#484
paradigmxyz/reth#10829

Also added hook for `eofcreate` and moved all handling away from `_env`
hooks to ensure correct ordering.

We don't yet insert logs for selfdestructs, would need update on revm
side to pass journaled state into the hook
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants