- How to write tests
- Test design
- a. ibctest
- b. CI configuration
- Github Workflows
- Running Compatibility Tests
- Troubleshooting
All tests should go under the e2e directory. When adding a new test, either add a new test function
to an existing test suite in the same file, or create a new test suite in a new file and add test functions there.
New test files should follow the convention of module_name_test.go.
New test suites should be composed of testsuite.E2ETestSuite. This type has lots of useful helper functionality that will
be quite common in most tests.
Note: see here for details about these requirements.
Tests can be run using a Makefile target under the e2e directory. e2e/Makefile
There are several envinronment variables that alter the behaviour of the make target.
| Environment Variable | Description | Default Value |
|---|---|---|
| CHAIN_IMAGE | The image that will be used for the chain | ibc-go-simd |
| CHAIN_A_TAG | The tag used for chain B | latest |
| CHAIN_B_TAG | The tag used for chain A | latest |
| CHAIN_BINARY | The binary used in the container | simd |
| RLY_TAG | The tag used for the go relayer | main |
Note: when running tests locally, no images are pushed to the
ghcr.io/cosmos/ibc-go-simdregistry. The images which are used only exist on your machine.
These environment variables allow us to run tests with arbitrary verions (from branches or released) of simd and the go relayer.
Every time changes are pushed to a branch or to main, a new simd image is built and pushed here.
export CHAIN_IMAGE="ghcr.io/cosmos/ibc-go-simd"
export CHAIN_A_TAG="main"
export CHAIN_BINARY="simd"
# We can also specify different values for the chains if needed.
# they will default to the same as chain a.
# export CHAIN_B_TAG="main"
# export CHAIN_BINARY="icad"
export RLY_TAG="v2.0.0"
make e2e-test entrypoint=TestInterchainAccountsTestSuite test=TestMsgSubmitTx_SuccessfulTransferNote: sometimes it can be useful to make changes to ibctest when running tests locally. In order to do this, add the following line to e2e/go.mod
replace github.com/strangelove-ventures/ibctest => ../ibctest
Or point it to any local checkout you have.
To run tests in CI, you can checkout the ibc-go repo and provide these environment variables to the CI task.
This repo contains an example of how to do this with Github Actions.
Every standard test will start with this. This creates two chains and a relayer, initializes relayer accounts on both chains, establishes a connection and a channel between the chains.
Both chains have started, but the relayer is not yet started.
The relayer should be started as part of the test if required. See Starting the Relayer
relayer, channelA := s.SetupChainsRelayerAndChannel(ctx, feeMiddlewareChannelOptions())
chainA, chainB := s.GetChains()There are helper functions to easily create users on both chains.
chainAWallet := s.CreateUserOnChainA(ctx, testvalues.StartingTokenAmount)
chainBWallet := s.CreateUserOnChainB(ctx, testvalues.StartingTokenAmount)We can wait for some number of blocks on the specified chains if required.
chainA, chainB := s.GetChains()
err := test.WaitForBlocks(ctx, 1, chainA, chainB)
s.Require().NoError(err)We can fetch balances of wallets on specific chains.
chainABalance, err := s.GetChainANativeBalance(ctx, chainAWallet)
s.Require().NoError(err)We can broadcast arbitrary messages which are signed on behalf of users created in the test.
This example shows a multi message transaction being broadcast on chainA and signed on behalf of chainAWallet.
relayer, channelA := s.SetupChainsRelayerAndChannel(ctx, feeMiddlewareChannelOptions())
chainA, chainB := s.GetChains()
chainAWallet := s.CreateUserOnChainA(ctx, testvalues.StartingTokenAmount)
chainBWallet := s.CreateUserOnChainB(ctx, testvalues.StartingTokenAmount)
t.Run("broadcast multi message transaction", func(t *testing.T){
payPacketFeeMsg := feetypes.NewMsgPayPacketFee(testFee, channelA.PortID, channelA.ChannelID, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), nil)
transferMsg := transfertypes.NewMsgTransfer(channelA.PortID, channelA.ChannelID, transferAmount, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), chainBWallet.Bech32Address(chainB.Config().Bech32Prefix), clienttypes.NewHeight(1, 1000), 0)
resp, err := s.BroadcastMessages(ctx, chainA, chainAWallet, payPacketFeeMsg, transferMsg)
s.AssertValidTxResponse(resp)
s.Require().NoError(err)
})The relayer can be started with the following.
t.Run("start relayer", func(t *testing.T) {
s.StartRelayer(relayer)
})Arbitrary commands can be executed on a given chain.
Note: these commands will be fully configured to run on the chain executed on (home directory, ports configured etc.)
However, it is preferable to broadcast messages or use a gRPC query if possible.
stdout, stderr, err := chainA.Exec(ctx, []string{"tx", "..."}, nil)It is possible to send an IBC transfer in two ways.
Use the ibctest Chain interface (this ultimately does a docker exec)
t.Run("send IBC transfer", func(t *testing.T) {
chainATx, err = chainA.SendIBCTransfer(ctx, channelA.ChannelID, chainAWallet.KeyName, walletAmount, nil)
s.Require().NoError(err)
s.Require().NoError(chainATx.Validate(), "chain-a ibc transfer tx is invalid")
})Broadcast a MsgTransfer.
t.Run("send IBC transfer", func(t *testing.T){
transferMsg := transfertypes.NewMsgTransfer(channelA.PortID, channelA.ChannelID, transferAmount, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), chainBWallet.Bech32Address(chainB.Config().Bech32Prefix), clienttypes.NewHeight(1, 1000), 0)
resp, err := s.BroadcastMessages(ctx, chainA, chainAWallet, transferMsg)
s.AssertValidTxResponse(resp)
s.Require().NoError(err)
})These E2E tests use the ibctest framework. This framework creates chains and relayers in containers and allows for arbitrary commands to be executed in the chain containers, as well as allowing us to broadcast arbitrary messages which are signed on behalf of a user created in the test.
There are two main github actions for e2e tests.
e2e.yaml which runs when collaborators create branches.
e2e-fork.yaml which runs when forks are created.
In e2e.yaml, the simd image is built and pushed to a registry and every test
that is run uses the image that was built.
In e2e-fork.yaml, images are not pushed to this registry, but instead remain local to the host runner.
The tests use the matrix feature of Github Actions. The matrix is
dynamically generated using this command.
Note: there is currently a limitation that all tests belonging to a test suite must be in the same file. In order to support test functions spread in different files, we would either need to manually maintain a matrix or update the script to account for this. The script assumes there is a single test suite per test file to avoid an overly complex generation process.
Which looks under the e2e directory, and creates a task for each test suite function.
// e2e/file_one_test.go
package e2e
func TestFeeMiddlewareTestSuite(t *testing.T) {
suite.Run(t, new(FeeMiddlewareTestSuite))
}
type FeeMiddlewareTestSuite struct {
testsuite.E2ETestSuite
}
func (s *FeeMiddlewareTestSuite) TestA() {}
func (s *FeeMiddlewareTestSuite) TestB() {}
func (s *FeeMiddlewareTestSuite) TestC() {}// e2e/file_two_test.go
package e2e
func TestTransferTestSuite(t *testing.T) {
suite.Run(t, new(TransferTestSuite))
}
type TransferTestSuite struct {
testsuite.E2ETestSuite
}
func (s *TransferTestSuite) TestD() {}
func (s *TransferTestSuite) TestE() {}
func (s *TransferTestSuite) TestF() {}In the above example, the following would be generated.
{
"include": [
{
"entrypoint": "TestFeeMiddlewareTestSuite",
"test": "TestA"
},
{
"entrypoint": "TestFeeMiddlewareTestSuite",
"test": "TestB"
},
{
"entrypoint": "TestFeeMiddlewareTestSuite",
"test": "TestC"
},
{
"entrypoint": "TestTransferTestSuite",
"test": "TestD"
},
{
"entrypoint": "TestTransferTestSuite",
"test": "TestE"
},
{
"entrypoint": "TestTransferTestSuite",
"test": "TestF"
}
]
}This string is used to generate a test matrix in the Github Action that runs the E2E tests.
All tests will be run on different hosts.
- Gas fees are set to zero to simply calcuations when asserting account balances.
- When upgrading from e.g. v4 -> v5, in ibc-go, we cannot upgrade the go.mod under
e2esince v5 will not yet exist. We need to upgrade it in a follow up PR.
If we ever need to manually build and push an image, we can do so with the Build Simd Image GitHub workflow.
This can be triggered manually from the UI by navigating to
Actions -> Build Simd Image -> Run Workflow
And providing the git tag.
Alternatively, the gh CLI tool can be used to trigger this workflow.
gh workflow run "Build Simd Image" -f tag=v3.0.0A full matrix of tests can be configured in json format. See this file as a reference.
To run all of the tests specified in this file, run
make compatibility-testsThis will run a GitHub Action for each entry, and display markdown which can be used in GitHub issue bodies to provide links to each of the workflows.
Note: a version field which corresponds to a subdirectory under scripts/test-matricies can also be specified. Once a file exists there,
e.g. scripts/test-matricies/v5.0.0/test-matrix.json, that file will be used to provide values to the GitHub workflows
if we run make compatibility-tests version=v5.0.0
-
On Mac, after running a lot of tests, it can happen that containers start failing. To fix this, you can try clearing existing containers and restarting the docker daemon.
This generally manifests itself as relayer or simd containers timing out during setup stages of the test. This doesn't happen in CI.
# delete all images docker system prune -afThis issue doesn't seem to occur on other operating systems.