Test Network Testing Framework #6489

alexanderbez · 2020-06-23T14:20:21Z

Description

Introduction of the testutil package. This package allows the creation of an entirely in-process testing cluster with fully operational Tendermint nodes constructed with SimApp. Each node has an RPC & API exposed. In addition, the network exposes a Local client that can be used to directly interface with Tendermint's RPC. The test network is entirely configurable. In the future, we should explore how to pass dynamic app constructors so other applications/projects can utilize this.

This is the foundation of #6423. With this framework, we'll be able to easily spin up and tear down networks for rapid integration testing and development. Once we refactor our CLI handling, we can entirely ditch all the current "fixtures".

I've included a "liveness" test as an example and replacement of the ad-hoc script we have. I've also included an integration test suite for x/bank API handlers.

/cc @aaronc @jackzampolin @anilcse @marbar3778

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer

alexanderbez · 2020-06-23T14:22:04Z

server/start.go

-		if err := apiSrv.Start(config); err != nil {
+		errCh := make(chan error)
+
+		go func() {
+			if err := apiSrv.Start(config); err != nil {
+				errCh <- err
+			}
+		}()
+
+		select {
+		case err := <-errCh:
 			return err
+		case <-time.After(5 * time.Second): // assume server started successfully


I did not realize Tendermint HTTP client actually blocks -- this fixes that so we can properly cleanup below.

codecov · 2020-06-23T14:44:31Z

Codecov Report

Merging #6489 into master will increase coverage by 4.51%.
The diff coverage is 85.29%.

@@            Coverage Diff             @@
##           master    #6489      +/-   ##
==========================================
+ Coverage   52.17%   56.68%   +4.51%     
==========================================
  Files         199      478     +279     
  Lines       12155    28777   +16622     
==========================================
+ Hits         6342    16313    +9971     
- Misses       5433    11306    +5873     
- Partials      380     1158     +778

server/api/server.go

testutil/network.go

fedekunze · 2020-06-23T22:19:40Z

testutil/util.go

+	cdc.MustUnmarshalJSON(cfg.GenesisState[authtypes.ModuleName], &authGenState)
+
+	authGenState.Accounts = genAccounts
+	cfg.GenesisState[authtypes.ModuleName] = cdc.MustMarshalJSON(authGenState)


See comment.

alexanderbez · 2020-06-24T01:01:32Z

Failures are due to my apparent misunderstanding of Tendermint's multi-process capabilities -- specifically the RPC. It really is only meant to run in a single process, so starting multiple RPC clients is not going to work :-/

The only stopgap I can think of is to only allow one use of a network at a time and within a network, only one single validator's RPC server/client is started (which is OK).

This might be an ok tradeoff for now.

alexanderbez · 2020-06-24T14:37:10Z

Still getting race conditions in Tendermint :-/

==================
WARNING: DATA RACE
Read at 0x00c000e22b30 by goroutine 166:
  github.com/tendermint/tendermint/p2p.(*NetAddress).String()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/netaddress.go:168 +0x4e
  github.com/tendermint/tendermint/p2p.(*NetAddress).Equals()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/netaddress.go:145 +0x5d
  github.com/tendermint/tendermint/p2p.(*Switch).IsPeerPersistent()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:589 +0xbd
  github.com/tendermint/tendermint/p2p.(*Switch).IsPeerPersistent-fm()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:587 +0x4b
  github.com/tendermint/tendermint/p2p.(*MultiplexTransport).wrapPeer()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/transport.go:491 +0x357
  github.com/tendermint/tendermint/p2p.(*MultiplexTransport).Accept()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/transport.go:200 +0x2f1
  github.com/tendermint/tendermint/p2p.(*Switch).acceptRoutine()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:598 +0x2a1

Previous write at 0x00c000e22b30 by goroutine 168:
  github.com/tendermint/tendermint/p2p.(*NetAddress).String()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/netaddress.go:173 +0xfb
  github.com/tendermint/tendermint/p2p.(*NetAddress).Equals()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/netaddress.go:145 +0x5d
  github.com/tendermint/tendermint/p2p.(*Switch).IsPeerPersistent()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:589 +0xbd
  github.com/tendermint/tendermint/p2p.(*Switch).IsPeerPersistent-fm()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:587 +0x4b
  github.com/tendermint/tendermint/p2p.(*MultiplexTransport).wrapPeer()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/transport.go:487 +0x72
  github.com/tendermint/tendermint/p2p.(*MultiplexTransport).Dial()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/transport.go:228 +0x2aa
  github.com/tendermint/tendermint/p2p.(*Switch).addOutboundPeerWithConfig()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:701 +0x45e
  github.com/tendermint/tendermint/p2p.(*Switch).DialPeerWithAddress()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:537 +0x1b5
  github.com/tendermint/tendermint/p2p.(*Switch).dialPeersAsync.func1()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:512 +0x2b3

Goroutine 166 (running) created at:
  github.com/tendermint/tendermint/p2p.(*Switch).OnStart()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/p2p/switch.go:234 +0x23a
  github.com/tendermint/tendermint/libs/service.(*BaseService).Start()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/libs/service/service.go:140 +0x504
  github.com/tendermint/tendermint/node.(*Node).OnStart()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/node/node.go:783 +0x605
  github.com/tendermint/tendermint/libs/service.(*BaseService).Start()
      /Users/aleksbez/go/pkg/mod/github.com/tendermint/tendermint@v0.33.5/libs/service/service.go:140 +0x504
  github.com/cosmos/cosmos-sdk/testutil.startInProcess()
      /Users/aleksbez/code/cosmos/cosmos-sdk/testutil/util.go:63 +0xa3b

I would imagine each node has its own unique address book (set of *NetAdress), so I'm not sure what's going on...

aaronc

Great work @alexanderbez ! Looks like a lot of heavy lifting.

I would like to have the configurability to use another app besides simapp for this functionality. I think that would basically mean having:

an app constructor on Config, and
test suites that have a constructor which takes a Config object

testutil/doc.go

testutil/network.go

testutil/util.go

Co-authored-by: Aaron Craelius <aaron@regen.network>

alexanderbez · 2020-06-25T23:53:21Z

Thanks @aaronc.

an app constructor on Config, and

Done.

test suites that have a constructor which takes a Config object

I don't follow. Can you elaborate, please?

…mos-sdk into bez/in-process-test-framework

alexanderbez · 2020-06-26T12:24:45Z

Bump @aaronc @AdityaSripal on a re-review & comment followup. I'd like to get this merged so I can start addressing the CLI testing refactor.

aaronc

@alexanderbez I added some TODOs to #6423 to track the TxGenerator migration and also to address reusability of IntegrationTestSuites. This looks like a great start.

aaronc · 2020-06-26T15:13:37Z

test suites that have a constructor which takes a Config object

I don't follow. Can you elaborate, please?

Basically we could have a NewIntegrationTestSuite(Config) constructor that allows for the IntegrationTestSuite to be used by different apps. We would also move IntegrationTestSuite to a helpers file so it can be used elsewhere. Then the default test runner code would look like:

	func TestIntegrationTestSuite(t *testing.T) {
 		suite.Run(t, new IntegrationTestSuite(testutil.DefaultConfig())
 	}

I added a TODO to #6423 about doing this in a separate PR.

alexanderbez · 2020-06-26T15:16:25Z

IMHO that might be overkill.

aaronc · 2020-06-26T15:38:24Z

IMHO that might be overkill.

What's overkill?

alexanderbez · 2020-06-26T15:44:05Z

#6489 (comment)

aaronc · 2020-06-26T15:53:52Z

#6489 (comment)

Why would that be overkill? Seems pretty simple and isn't one of the points to be able to reuse integration tests in gaia and other apps?

alexanderbez · 2020-06-26T16:06:22Z

IntegrationTestSuite isn't meant to be used by other apps. Each package should define its own test suite. Doing what you're suggesting doesn't really buy us anything valuable.

aaronc · 2020-06-26T16:17:22Z

I don’t think that’s true. Each integration test suite is scoped to a package but when we are testing an app such as gaia or regen, the app developer likely wants to verify that the app has correctly wired together the various modules. We are not deploying simapp to a live network, we are deploying other apps. For regen’s codebase most of the code is imported from other repos but l still want to have tests of that code wired together and the primary test suite l will want to run are these integration tests for each of the modules we’re importing running against the regen, not simapp, app constructor.

alexanderbez · 2020-06-26T16:30:22Z

Mhhh, ok I'm still not really following what IntegrationTestSuite buys you than you can't just do directly. In any case, I'm not going to block this PR on this. Feel free to open up a follow-up PR with this ask and I'll be happy to review.

Addressed all comments.

aaronc · 2020-06-26T16:38:42Z

Mhhh, ok I'm still not really following what IntegrationTestSuite buys you than you can't just do directly. In any case, I'm not going to block this PR on this. Feel free to open up a follow-up PR with this ask and I'll be happy to review.

Yep, like I said it can be addressed later and I put a TODO in #6423.

alexanderbez added 2 commits June 23, 2020 10:15

Init commit

c9d5b34

Remove script

1e651e8

alexanderbez added T: Tests WIP labels Jun 23, 2020

Merge branch 'master' into bez/in-process-test-framework

9d8494f

alexanderbez commented Jun 23, 2020

View reviewed changes

alexanderbez mentioned this pull request Jun 23, 2020

In Process Integration Tests #6423

Closed

19 tasks

alexanderbez and others added 9 commits June 23, 2020 12:16

Fix IBC codec bug

cd2ec2d

go mod tidy + codec fixes attempt

ecfa3c8

Change base dir to use chain-id

c7e9b60

Add trap signal call

f5deb8d

Add godoc

7f51e12

Merge branch 'master' into bez/in-process-test-framework

b7c2cca

Updates

ec88212

Tweaks

4f43424

Fix fn name

e8f5176

anilcse reviewed Jun 23, 2020

View reviewed changes

server/api/server.go Outdated Show resolved Hide resolved

testutil/network.go Show resolved Hide resolved

Cleanup

433b67e

fedekunze reviewed Jun 23, 2020

View reviewed changes

alexanderbez added 6 commits June 23, 2020 22:54

Only allow 1st val to serve rpc/api

f7758f7

use pkg lock

4bc56bd

Merge branch 'master' into bez/in-process-test-framework

d430884

lint++

c389c81

lint++

a6f8fe3

Remove use of errgroup

9442a8b

ebuchman mentioned this pull request Jun 24, 2020

p2p: data race in NetAddress.String() memoization tendermint/tendermint#5047

Closed

ci: remove -race flag from tests

0aff4e4

aaronc requested changes Jun 25, 2020

View reviewed changes

testutil/doc.go Outdated Show resolved Hide resolved

testutil/doc.go Outdated Show resolved Hide resolved

testutil/doc.go Outdated Show resolved Hide resolved

testutil/network.go Show resolved Hide resolved

testutil/util.go Outdated Show resolved Hide resolved

alexanderbez and others added 3 commits June 25, 2020 19:49

Update testutil/doc.go

cdc9db6

Co-authored-by: Aaron Craelius <aaron@regen.network>

Update testutil/doc.go

8aa5c71

Co-authored-by: Aaron Craelius <aaron@regen.network>

Update testutil/doc.go

426505e

Co-authored-by: Aaron Craelius <aaron@regen.network>

alexanderbez added 6 commits June 25, 2020 20:20

testutil: make app confiruable

c6a208a

testutil: update go doc

05fd4a7

Merge branch 'bez/in-process-test-framework' of github.com:cosmos/cos…

38aac75

…mos-sdk into bez/in-process-test-framework

testutil: revise app con

195447e

testutil: update go doc

939b6c3

Fixes

9c77941

alexanderbez requested a review from aaronc June 26, 2020 00:32

Merge branch 'master' into bez/in-process-test-framework

89a447d

aaronc approved these changes Jun 26, 2020

View reviewed changes

Merge branch 'master' into bez/in-process-test-framework

df0b1b6

Merge branch 'master' into bez/in-process-test-framework

ddb342b

alexanderbez merged commit 9bf3ff7 into master Jun 26, 2020

alexanderbez deleted the bez/in-process-test-framework branch June 26, 2020 16:30

timlind mentioned this pull request Oct 22, 2021

Write in-process integration test for CLP create pool Sifchain/sifnode#2089

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Network Testing Framework #6489

Test Network Testing Framework #6489

alexanderbez commented Jun 23, 2020 •

edited

Loading

alexanderbez Jun 23, 2020 •

edited

Loading

codecov bot commented Jun 23, 2020 •

edited

Loading

fedekunze Jun 23, 2020

alexanderbez Jun 23, 2020

alexanderbez commented Jun 24, 2020

alexanderbez commented Jun 24, 2020 •

edited

Loading

aaronc left a comment •

edited

Loading

alexanderbez commented Jun 25, 2020 •

edited

Loading

alexanderbez commented Jun 26, 2020

aaronc left a comment

aaronc commented Jun 26, 2020 •

edited

Loading

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020 via email •

edited

Loading

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

Test Network Testing Framework #6489

Test Network Testing Framework #6489

Conversation

alexanderbez commented Jun 23, 2020 • edited Loading

Description

alexanderbez Jun 23, 2020 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jun 23, 2020 • edited Loading

Codecov Report

fedekunze Jun 23, 2020

Choose a reason for hiding this comment

alexanderbez Jun 23, 2020

Choose a reason for hiding this comment

alexanderbez commented Jun 24, 2020

alexanderbez commented Jun 24, 2020 • edited Loading

aaronc left a comment • edited Loading

Choose a reason for hiding this comment

alexanderbez commented Jun 25, 2020 • edited Loading

alexanderbez commented Jun 26, 2020

aaronc left a comment

Choose a reason for hiding this comment

aaronc commented Jun 26, 2020 • edited Loading

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020 via email • edited Loading

alexanderbez commented Jun 26, 2020

aaronc commented Jun 26, 2020

alexanderbez commented Jun 23, 2020 •

edited

Loading

alexanderbez Jun 23, 2020 •

edited

Loading

codecov bot commented Jun 23, 2020 •

edited

Loading

alexanderbez commented Jun 24, 2020 •

edited

Loading

aaronc left a comment •

edited

Loading

alexanderbez commented Jun 25, 2020 •

edited

Loading

aaronc commented Jun 26, 2020 •

edited

Loading

aaronc commented Jun 26, 2020 via email •

edited

Loading