Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tealdbg: increase intermediate reading/writing buffers #3335

Merged
merged 3 commits into from
Dec 20, 2021

Conversation

barnjamin
Copy link
Contributor

Summary

Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files.

closes #3100

Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

@codecov-commenter
Copy link

codecov-commenter commented Dec 20, 2021

Codecov Report

Merging #3335 (0d0e4c3) into master (9cdcb39) will increase coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3335      +/-   ##
==========================================
+ Coverage   47.64%   47.65%   +0.01%     
==========================================
  Files         369      369              
  Lines       59629    59629              
==========================================
+ Hits        28412    28418       +6     
+ Misses      27908    27904       -4     
+ Partials     3309     3307       -2     
Impacted Files Coverage Δ
cmd/tealdbg/server.go 35.48% <ø> (ø)
ledger/blockqueue.go 82.18% <0.00%> (-2.88%) ⬇️
network/wsPeer.go 68.05% <0.00%> (ø)
network/requestTracker.go 71.12% <0.00%> (+0.86%) ⬆️
crypto/merkletrie/node.go 93.48% <0.00%> (+1.86%) ⬆️
crypto/merkletrie/trie.go 68.61% <0.00%> (+2.18%) ⬆️
cmd/algoh/blockWatcher.go 80.95% <0.00%> (+3.17%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9cdcb39...0d0e4c3. Read the comment docs.

Copy link
Contributor

@tsachiherman tsachiherman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change makes sense; please define constants for these parameter, with their absolute values, and write in the description the rationale.

tsachiherman
tsachiherman previously approved these changes Dec 20, 2021
Copy link
Contributor

@tsachiherman tsachiherman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for the changes.

@tsachiherman
Copy link
Contributor

Could you please fix the reviewdog issues ?

@tsachiherman tsachiherman changed the title up buffer size for tealdbg tealdbg: increase intermediate reading/writing buffers Dec 20, 2021
@tsachiherman tsachiherman merged commit 0ad9900 into algorand:master Dec 20, 2021
jannotti added a commit that referenced this pull request Jan 7, 2022
* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed

Guessing from this change #3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* license check

* Shut up, dawg.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
jannotti added a commit that referenced this pull request Jan 7, 2022
* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed

Guessing from this change #3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* remove buildnumber.dat

* license check

* Shut up, dawg.

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves algorand/go-algorand-internal#1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>
jannotti added a commit that referenced this pull request Jan 7, 2022
* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed

Guessing from this change #3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* remove buildnumber.dat

* license check

* Shut up, dawg.

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves algorand/go-algorand-internal#1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>
jannotti added a commit that referenced this pull request Jan 7, 2022
* Update the Version, BuildNumber, genesistimestamp.data

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed

Guessing from this change #3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* remove buildnumber.dat

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves algorand/go-algorand-internal#1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>
jannotti added a commit that referenced this pull request Jan 7, 2022
* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed

Guessing from this change #3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* remove buildnumber.dat

* license check

* Shut up, dawg.

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves algorand/go-algorand-internal#1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>
jannotti added a commit that referenced this pull request Jan 18, 2022
* The new inner appl fields

* Unit tests for field setting on appls

* Construct EvalDelta in AVM rather than by inspecting ledger

* Obey the linter!

* more LedgerForEvaluation accomodation

* Test inner evaldeltas

* Checks on calling old AVM apps, or re-entrancy

* Allow opcode budget to be added to by executing inner apps.

* TxID and GroupID for inner transactions

* gitxn/gitxna

* Lint, spec generate

* txn simplifications

* Encode "arrayness" in the txn field spec

* Pavel's CR comments

* Update tests to distinguish assembly / eval errors

* Test itxn_field assembly separate from eval

* factor out the array index parsing of all the txn assembly

* Consistent errors and parsign for many opcodes

* Cleanup immediate parsing, prep txn effects testing

* EvalParams is now a single object used to evaluate each txn in turn.

* Simplifications for the Dawg (the Review Dog)

* Use a copy for the EvalParams.TxnGroup

* Set the logicsig on txns in the GroupContext, so check() can see it

* Update test for explicit empty check

* Three new globals for to help contract-to-contract usability (#3237)

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Gloadss (#3248)

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* opcode, docs, and tests

* specs update

* Feature/contract to contract (#3285)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>

* add access to resources created in the current group (#3340)

* Feature/contract to contract (#3357)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* license check

* Shut up, dawg.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>

* Feature/contract to contract (#3389)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* remove buildnumber.dat

* license check

* Shut up, dawg.

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves https://github.com/algorand/go-algorand-internal/issues/1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>

* Feature/contract to contract (#3390)

* Update the Version, BuildNumber, genesistimestamp.data

* Thr…
@algobarb algobarb mentioned this pull request Jan 18, 2022
jannotti added a commit that referenced this pull request Feb 17, 2022
* The new inner appl fields

* Unit tests for field setting on appls

* Construct EvalDelta in AVM rather than by inspecting ledger

* Obey the linter!

* more LedgerForEvaluation accomodation

* Test inner evaldeltas

* Checks on calling old AVM apps, or re-entrancy

* Allow opcode budget to be added to by executing inner apps.

* TxID and GroupID for inner transactions

* gitxn/gitxna

* Lint, spec generate

* txn simplifications

* Encode "arrayness" in the txn field spec

* Pavel's CR comments

* Update tests to distinguish assembly / eval errors

* Test itxn_field assembly separate from eval

* factor out the array index parsing of all the txn assembly

* Consistent errors and parsign for many opcodes

* Cleanup immediate parsing, prep txn effects testing

* EvalParams is now a single object used to evaluate each txn in turn.

* Simplifications for the Dawg (the Review Dog)

* Use a copy for the EvalParams.TxnGroup

* Set the logicsig on txns in the GroupContext, so check() can see it

* Update test for explicit empty check

* Three new globals for to help contract-to-contract usability (#3237)

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Gloadss (#3248)

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* opcode, docs, and tests

* specs update

* Feature/contract to contract (#3285)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>

* add access to resources created in the current group (#3340)

* Feature/contract to contract (#3357)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* license check

* Shut up, dawg.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>

* Feature/contract to contract (#3389)

* Update the Version, BuildNumber, genesistimestamp.data

* Three new globals for to help contract-to-contract usability

* detritis

* Check error

* doc comments

* Support transaction arguments for `goal app method` (#3233)

* Implement transactions as arguments

* Fix indexing and dryrun issue

* Add docstring

* Satisfy review dog

* Fix pointer issue

* Fix group command

* Rename e2e test

* Fix filename variable

* Add e2e test

* Use tab

* CI: use libboost-math-dev instead of libboost-all-dev (#3223)

## Summary

Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.)

## Test Plan

Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern.

* testing: fixes to rest-participation-key e2e test (#3238)

## Summary

- Test to make sure RES has the right input before counting line numbers for result size.
- Rest RES to empty so that the same output is not recycled in case of an error.
- exit 1 in case of an error
- Reduce LAST_ROUND from 1200000 to 120
- "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value.

* testing: interactive mode for e2e testing (#3227)



## Summary
Some e2e tests require a python environment for testing.
Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial.
This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment.


## Test Plan
No tests needed. Tested the script locally.

* Make dev-mode tests less flaky. (#3252)

## Summary

Fix a couple flaws in the new go-e2e tests built ontop of DevMode:
* Shutdown the fixture when finished.
* Don't run in parallel.
* Longer delays / better algorithms to wait for data flushing to complete.
* Check for "out of order" keys.

## Test Plan

N/A, this is a test.

* adding libtool to ubuntu deps (#3251)

## Summary

The sandbox is not building with dev config using master branch https://github.com/algorand/sandbox/issues/85, complains about libtool not being installed

Guessing from this change https://github.com/algorand/go-algorand/pull/3223 

Adding libtool to UBUNTU_DEPS in install scripts

## Test Plan

Set config in sandbox to my branch
`sandbox up dev`
It built

* Fix error shadowing in Eval (#3258)

## Summary

Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure.

## Test Plan

Use existing tests

* Disable flaky test. (#3256)

## Summary

This test doesn't work properly, disable it until #3255 addresses any underlying problems.

* Update the Version, BuildNumber, genesistimestamp.data

* Fix a data race in app tests (#3269)

## Summary

A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed.

## Test Plan

```
go test ./ledger -run TestAppEmpty -race -count=50
ok      github.com/algorand/go-algorand/ledger  4.078s
```

* Fix e2e.sh mixed indent characters. (#3266)

## Summary

Fix e2e.sh mixed indent characters.

* Fix ParticipationKeyResponse type. (#3264)

## Summary

Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen.

* disable parallelism for e2e-go tests (#3242)

## Summary

This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention.

## Test Plan

Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run.

* Updating Readme.md with circleci status badges (#3245)

* Fix formatting for CircleCI badges (#3272)

* Add Custom Scenario for Performance Testing (#3278)

Add Custom Scenario for Performance Testing.
Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe.

* Impose limits on the entire "tree" of inner calls.

This also increases the realism of testing of multiple app calls in a
group by creating the EvalParams with the real constructor, thus
getting the pooling stuff tested here without playing games
manipulating the ep after construction.

* ParticipationRegistry - StateProof loading methods (#3261)

## Summary

Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later.
 
## Test Plan

New unit tests.

* Op base64 decode (#3220)

b64 opcode, tests, and specs

* Bump Version, Remove buildnumber.dat and genesistimestamp.dat files.

* Change golang version to 1.16.11 in go-algorand (#2825)

Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version.
Release notes for Go 1.15 and 1.16:
https://tip.golang.org/doc/go1.16
https://tip.golang.org/doc/go1.15

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* TestEcdsa: fix flaky "tampering" of public key (#3282)

## Summary

This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`)

## Test Plan

Existing test should pass, occasional flakiness should go away.

* Move appID tracking into EvalContext, out of LedgerForLogic

This change increases the seperation between AVM execution and the
ledger being used to lookup resources.  Previously, the ledger kept
track of the appID being executed, to offer a narrower interface to
those resources. But now, with app-to-app calls, the appID being
executed must change, and the AVM needs to maintain the current appID.

* Stupid linter

* Support reference types in `goal app method` (#3275)

* Fix method signature parse bug

* Support reference types

* Review dog fixes

* Fix comments

* Add a hash prefix for ARCs-related hashes (#3298)

## Summary

This is to allow ARCs (github.com/algorandfoundation/ARCs)
to have their own hash prefix without risk of collision.

## Test Plan

It is purely informational. There is no real code change.

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Compatibility mode for partkeyinfo. (#3291)

## Summary

Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running

Workarounds:
1) use an older `goal` binary.
2) use `algokey part info --keyfile <file>`

## Test Plan

Tested manually:
```
~$ goal account partkeyinfo -d /tmp/private_network/Node/
Dumping participation key info from /tmp/private_network/Node/...

Participation ID:          CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA
Parent address:            DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU
Last vote round:           1
Last block proposal round: 2
Effective first round:     1
Effective last round:      3000000
First round:               0
Last round:                3000000
Key dilution:              10000
Selection key:             5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=
Voting key:                PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=

~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c
Dumping participation key info from /tmp/private_network/Node/...
------------------------------------------------------------------
File: Wallet2.0.3000000.partkey
{
  "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU",
  "last": 3000000,
  "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=",
  "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=",
  "voteKD": 10000
}
```

* catchup: suspend the catchup session once the agreement service kicks in (#3299)

The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on.

The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it.
In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete.

When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. 

In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. 
Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup.

A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch.
However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete.
This change fixes the return value from true to false.

* Bump buildnumber.dat

* testing: disable flaky test (#3268)

Disable a flaky test, to be re-enabled later with #3267.

* enumerate conditions that might cause this fetchAndWrite to return false (#3301)

## Summary

The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up.  The conditions that might cause it to return false should be more explicitly enumerated.

## Test Plan

Just comments

* Fix unit tests error messages

* make sure the block service is not attempting to access the ledger after being stopped. (#3303)

## Summary

The block service was attempting to serve block via the http handler even after it has been stopped.
This lead to undesired downstream failures in the ledger, which was shutdown as well.

## Test Plan

unit test added.

* Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308)

## Summary

Avoid creating algod process for the sole purpose of retrieving the genesis-id.

Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string.
The genesis version string can be easily retrieved by loading the genesis file.

## Test Plan

Use existing e2e tests.

* documentation: fix algorand specs link (#3309)

## Summary
This PR fixes a link in a README.

## Testing
I clicked on the new link.

* testing: reword partitiontest lint message. (#3297)

## Summary

The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets).

* testing: fix random data race in TestAppAccountDataStorage (#3315)

fix random data race in unit test

* Allow access to resources created in the same transaction group

The method will be reworked, but the tests are correct and want to get
them visible to team.

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Separate tx and key validity for `goal account renewpartkey` (#3286)

Always use currentRound+proto.MaxTxnLife as last valid round for the
transaction when renewing instead of using the partkey validity period.

This fixes #3283

* Add qkniep to THANKS.md (#3320)

## Summary

Add qkniep to THANKS.md

* Followup to opcode base64_decode (#3288)

* alphabet begone in favor of encoding

* unit test various padding and whitespace scenarios

* padding permutations also fail

* "Slicing" --> "Manipulation"

* fix the codegen fail?

* Documenting padding, whitespace, other character behavior

* Add help and fish mode to e2e interactive mode. (#3313)

## Summary

Minor improvements to e2e.sh interactive mode:
* add to -h output
* do not run start stop test in interactive mode
* support fish shell

## Test Plan

Manual testing:
```
~$ ./e2e.sh -i
...
lots of output removed
...
********** READY **********


The test environment is now set. You can now run tests in another terminal.

Configure the environment:

set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve"
set -g PATH "$VIRTUAL_ENV/bin:$PATH"

python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py  "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME

Press enter to shut down the test environment...
```

* Minimum Account Balance in Algod (#3287)

* Access to apps created in group

Also adds some tests that are currently skipped for testing
- access to addresses of newly created apps
- use of gaid in inner transactions

Both require some work to implement the thing being tested.

* Remove tracked created mechanism in favor of examining applydata.

* Add convertAddress tool. (#3304)

## Summary

New tool: convertAddress
I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful.

Merge `debug` into `tools` to make it easier to organize these miscellaneous tools.

* tealdbg: increase intermediate reading/writing buffers (#3335)

## Summary

Some large teal source files cause the tealdbg/cdt session to choke.  Upping the buffer size to allow for larger source files.

closes #3100 

## Test Plan

Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking.

* Adding method pseudo op to readme (#3338)

* Allow v6 AVM code to use in-group created asas, apps (& their accts)

One exception - apps can not mutate (put or del) keys from the app
accounts, because EvalDelta cannot encode such changes.

* lint docs

* typo

* The review dog needs obedience training.

* add config.DeadlockDetectionThreshold (#3339)

Summary
This allows for the deadlock detection threshold to be set by configuration.

Test Plan
Existing tests should pass.

* Use one EvalParams for logic evals, another for apps in dry run

We used to use one ep per transaction, shared between sig and and
app. But the new model of ep usage is to keep using one while
evaluating an entire group.

The app ep is now built logic.NewAppEvalParams which, hopefully, will
prevent some bugs when we change something in the EvalParams and don't
reflect it in what was a "raw" EvalParams construction in debugger and
dry run.

* Use logic.NewAppEvalParams to decrease copying and bugs in debugger

* Simplify use of NewEvalParams. No more nil return when no apps.

This way, NewEvalParams can be used for all creations of EvalParams,
whether they are intended for logicsig or app use, greatly simplifying
the way we make them for use by dry run or debugger (where they serve
double duty).

* Remove explicit PastSideEffects handling in tealdbg

* Fix flaky test in randomized ABI encoding test (#3346)

* update abi encoding test random testcase generator, scale down parameters to avoid flaky test

* parameterized test script

* add notes to explain why flaky test is eliminated

* show more information from self-roundtrip testing

* fully utilize require, remove fmt

* Always create EvalParams to evaluate a transaction group.

We used to have an optimization to avoid creating EvalParams unless
there was an app call in the transaction group.  But the interface to
allow transaction processing to communicate changes into the
EvalParams is complicated by that (we must only do it if there is
one!)

This also allows us to use the same construction function for eps
created for app and logic evaluation, simplifying dry-run and
debugger.

The optimization is less needed now anyway:
1) The ep is now shared for the whole group, so it's only one.
2) The ep is smaller now, as we only store nil pointers instead of
larger scratch space objects for non-app calls.

* Correct mistaken commit

* ledger: perform the catchpoint writing outside the trackers lock. (#3311)

## Summary

This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes.

## Test Plan

* [x] Test manually & use existing tests.
* [x] Implement a unit test
* [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing.

* Bump version number

* Update `goal app method` handling of args and return values (#3352)

* Fix method call arg overflow handling

* Only check last log for return value

* Address feedback

* Add comment explaining ABI return prefix

* Support app creation in `goal app method` (#3353)

* Support app creation in `goal app method`

* Don't use nonprintable tab character

* Link to specific gist version

* Fix error messages

* Rename `methodCreation` to `methodCreatesApp`

* Spec improvments

* Update license to 2022 (#3360)

Update license on all source files to 2022.

* Add totals checks into acct updates tests (#3367)

## Summary

After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts.

## Test Plan

This is tests update

* More spec improvments, including resource "availability"

* Recursively return inner transaction tree

* Lint

* No need for ConfirmedRound, so don't deref a nil pointer!

* remove buildnumber.dat

* license check

* Shut up, dawg.

* PKI State Proof Incremental Key Loading (#3281)

## Summary

Followup to #3261 (contained in diff).

Use the new key loading routine from the REST API.

## Test Plan

New unit tests.

* Limit number of simultaneous REST connections (#3326)

## Summary

This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash.

Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed.

Partially resolves https://github.com/algorand/go-algorand-internal/issues/1814.

## Test Plan

Added unit tests.

* Use rejecting limit listener in WebsocketNetwork. (#3380)

## Summary

Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster.

## Test Plan

Probably not necessary.

* Delete unused constant. (#3379)

## Summary

This PR deletes an unused constant.

## Test Plan

None.

* Add test to exercise lookup corner cases (#3376)

## Summary

This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`).

In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before.

## Test Plan

Adds new test.

* Test for catchup stop on completion (#3306)

Adding a test for the fix in #3299

## Test Plan

This is a test

* Delete unused AtomicCommitWriteLock(). (#3383)

## Summary

This PR deletes unused `AtomicCommitWriteLock()` and simplifies code.

## Test Plan

None.

Co-authored-by: DevOps Service <devops-service@algorand.com>
Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com>
Co-authored-by: chris erway <51567+cce@users.noreply.github.com>
Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com>
Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com>
Co-authored-by: Will Winder <wwinder.unh@gmail.com>
Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com>
Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com>
Co-authored-by: Jack <87339414+algojack@users.noreply.github.com>
Co-authored-by: John Lee <john.lee@algorand.com>
Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com>
Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>
Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org>
Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com>
Co-authored-by: Tolik Zinovyev <tolik@algorand.com>
Co-authored-by: egieseke <eric_gieseke@yahoo.com>
Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de>
Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com>
Co-authored-by: Or Aharonee <or.aharonee@algorand.com>
Co-authored-by: Barbara Poon <barbara.poon@algorand.com>

* Feature/contract to contract (#3390)

* Update the Version, BuildNumber, genesistimestamp.dat…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tealdbg: large program breaks cdt session
4 participants