Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Update the Version, BuildNumber, genesistimestamp.data * Support transaction arguments for `goal app method` (#3233) * Implement transactions as arguments * Fix indexing and dryrun issue * Add docstring * Satisfy review dog * Fix pointer issue * Fix group command * Rename e2e test * Fix filename variable * Add e2e test * Use tab * CI: use libboost-math-dev instead of libboost-all-dev (#3223) ## Summary Small change: libboost-math-dev requires just 4 packages to install, while libboost-all-dev requires > 100. Only Debian/Ubuntu distributions provide fine-grained boost packages like this, but should shave a little time off the CI builds. (Our only boost include is boost/math/distributions/binomial.hpp.) ## Test Plan Builds should pass as before. Now that we are no longer using Travis for Linux builds, the side effect of libboost-all-dev installing make and other missing build tools on Travis encountered in #2717 is no longer a concern. * testing: fixes to rest-participation-key e2e test (#3238) ## Summary - Test to make sure RES has the right input before counting line numbers for result size. - Rest RES to empty so that the same output is not recycled in case of an error. - exit 1 in case of an error - Reduce LAST_ROUND from 1200000 to 120 - "Get List of Keys" before getting NUM_IDS_3 otherwise it will recycle old RES value. * testing: interactive mode for e2e testing (#3227) ## Summary Some e2e tests require a python environment for testing. Unfortunately, setting up that environment adequately similar to the testing environment may not be trivial. This change introduces an interactive mode to the e2e.sh script which stops at the point of running the tests, and allows the user to run the tests from the same testing environment. ## Test Plan No tests needed. Tested the script locally. * Make dev-mode tests less flaky. (#3252) ## Summary Fix a couple flaws in the new go-e2e tests built ontop of DevMode: * Shutdown the fixture when finished. * Don't run in parallel. * Longer delays / better algorithms to wait for data flushing to complete. * Check for "out of order" keys. ## Test Plan N/A, this is a test. * adding libtool to ubuntu deps (#3251) ## Summary The sandbox is not building with dev config using master branch algorand/sandbox#85, complains about libtool not being installed Guessing from this change #3223 Adding libtool to UBUNTU_DEPS in install scripts ## Test Plan Set config in sandbox to my branch `sandbox up dev` It built * Fix error shadowing in Eval (#3258) ## Summary Error from account preloading was shadowed by returning a wrong err variable. This caused subsequent problems in account updates and masked the original failure. ## Test Plan Use existing tests * Disable flaky test. (#3256) ## Summary This test doesn't work properly, disable it until #3255 addresses any underlying problems. * Update the Version, BuildNumber, genesistimestamp.data * Fix a data race in app tests (#3269) ## Summary A test helper function `commitRound` accessed `l.trackers.lastFlushTime` without taking a lock. Fixed. ## Test Plan ``` go test ./ledger -run TestAppEmpty -race -count=50 ok github.com/algorand/go-algorand/ledger 4.078s ``` * Fix e2e.sh mixed indent characters. (#3266) ## Summary Fix e2e.sh mixed indent characters. * Fix ParticipationKeyResponse type. (#3264) ## Summary Fix a small type discrepancy in the OpenAPI spec ahead of some other work that's about to happen. * disable parallelism for e2e-go tests (#3242) ## Summary This sets `-p 1` for the e2e-go tests, intended to make them more deterministic when running on a VM with relatively constrained resources. Since each e2e-go test might spin up a few nodes, it seems like it would help to avoid resource contention. ## Test Plan Tests should run as before. Desired effect can be verified by looking at the test output where the value of PARALLEL_FLAG is printed out before tests are run. * Updating Readme.md with circleci status badges (#3245) * Fix formatting for CircleCI badges (#3272) * Add Custom Scenario for Performance Testing (#3278) Add Custom Scenario for Performance Testing. Add README on how to run custom scenario and modify create_and_deploy_recipe.sh to accept a network template that will generate a new recipe. * ParticipationRegistry - StateProof loading methods (#3261) ## Summary Add ParticipationRegistry methods for setting and retrieving state proof keys. Since they aren't in master yet there is a `type StateProofKey []byte` stub which will need to be updated later. ## Test Plan New unit tests. * Op base64 decode (#3220) b64 opcode, tests, and specs * Bump Version, Remove buildnumber.dat and genesistimestamp.dat files. * Change golang version to 1.16.11 in go-algorand (#2825) Upgrading to 1.16 to help alleviate issues with working on different go versions, and update to a supported, more secure version. Release notes for Go 1.15 and 1.16: https://tip.golang.org/doc/go1.16 https://tip.golang.org/doc/go1.15 * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * TestEcdsa: fix flaky "tampering" of public key (#3282) ## Summary This test (TestEcdsa) tests the ecdsa_pk_decompress opcode and intentionally "tampers" with the public key by setting the first byte to zero. Occasionally this test is failing, likely because the first byte was already zero. (The test failures are for the cases where failure is expected, `pass=false`) ## Test Plan Existing test should pass, occasional flakiness should go away. * Support reference types in `goal app method` (#3275) * Fix method signature parse bug * Support reference types * Review dog fixes * Fix comments * Add a hash prefix for ARCs-related hashes (#3298) ## Summary This is to allow ARCs (github.com/algorandfoundation/ARCs) to have their own hash prefix without risk of collision. ## Test Plan It is purely informational. There is no real code change. * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Compatibility mode for partkeyinfo. (#3291) ## Summary Compatibility for `partkeyinfo` was also needed by some users. In addition to the different format, the old command also allows printing key information when the node is not running Workarounds: 1) use an older `goal` binary. 2) use `algokey part info --keyfile <file>` ## Test Plan Tested manually: ``` ~$ goal account partkeyinfo -d /tmp/private_network/Node/ Dumping participation key info from /tmp/private_network/Node/... Participation ID: CPLHRU3WEY3PE7XTPPSIE7BGJYWAIFPS7DL3HZNC4OKQRQ5YAYUA Parent address: DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU Last vote round: 1 Last block proposal round: 2 Effective first round: 1 Effective last round: 3000000 First round: 0 Last round: 3000000 Key dilution: 10000 Selection key: 5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ= Voting key: PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs= ~$ goal account partkeyinfo -d /tmp/private_network/Node/ -c Dumping participation key info from /tmp/private_network/Node/... ------------------------------------------------------------------ File: Wallet2.0.3000000.partkey { "acct": "DGS6VNX2BRMKGKVAS2LTREMYG33TOCYPFLPCQ3DUTJULQU6P6S7KJCDNTU", "last": 3000000, "sel": "5QRrTgzSUTqqym43QVsBus1/AOwGR5zE+I7FGwA14vQ=", "vote": "PK0NMyZ4BKSjPQ9JuT7dQBLdTpjLQv2txuDYDKhkuqs=", "voteKD": 10000 } ``` * catchup: suspend the catchup session once the agreement service kicks in (#3299) The catchup service stops when it is complete, i.e. it has reached up to the round which is being agreed on. The catchup service knows it is complete and should stop, when it finds that a block is in the ledger before it adds it. In other words, apart from the catchup, only the agreement adds blocks to the ledger. And when the agreement adds a block to the ledger before the catchup, it means the agreement is ahead, and the catchup is complete. When `fetchAndWrite` detects the block is already in the ledger, it returns. The return value of `false` stops the catchup syncing. In previous releases, `fetchAndWrite` was only checking if the block is already in the ledger after attempting to fetch it. Since it fails to fetch a block not yet agreed on, the fetch fails after multiple attempts, and `fetchAndWrite` returns `false` ending the catchup. A recent change made this process more efficient by first checking if the block is in the ledger before/during the fetch. However, once the block was found in the ledger, `fetchAndWrite` returned true instead of false (consistent with already existing logic since forever, which was also wrong). This caused the catchup to continue syncing after catchup was complete. This change fixes the return value from true to false. * Bump buildnumber.dat * testing: disable flaky test (#3268) Disable a flaky test, to be re-enabled later with #3267. * enumerate conditions that might cause this fetchAndWrite to return false (#3301) ## Summary The fetchAndWrite function contains some complex logic to ultimately determine if we should continue trying to catch up. The conditions that might cause it to return false should be more explicitly enumerated. ## Test Plan Just comments * make sure the block service is not attempting to access the ledger after being stopped. (#3303) ## Summary The block service was attempting to serve block via the http handler even after it has been stopped. This lead to undesired downstream failures in the ledger, which was shutdown as well. ## Test Plan unit test added. * Avoid creating algod process for the sole purpose of retrieving the genesis-id. (#3308) ## Summary Avoid creating algod process for the sole purpose of retrieving the genesis-id. Existing code was calling `algod -G -d <data dir>` in order to obtain the genesis version string. The genesis version string can be easily retrieved by loading the genesis file. ## Test Plan Use existing e2e tests. * documentation: fix algorand specs link (#3309) ## Summary This PR fixes a link in a README. ## Testing I clicked on the new link. * testing: reword partitiontest lint message. (#3297) ## Summary The wording on this was tripping me, maybe I was having an off day. I think it would be slightly easier if the message were to tell exactly what you need to do (and not use the angle brackets). * testing: fix random data race in TestAppAccountDataStorage (#3315) fix random data race in unit test * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Separate tx and key validity for `goal account renewpartkey` (#3286) Always use currentRound+proto.MaxTxnLife as last valid round for the transaction when renewing instead of using the partkey validity period. This fixes #3283 * Add qkniep to THANKS.md (#3320) ## Summary Add qkniep to THANKS.md * Followup to opcode base64_decode (#3288) * alphabet begone in favor of encoding * unit test various padding and whitespace scenarios * padding permutations also fail * "Slicing" --> "Manipulation" * fix the codegen fail? * Documenting padding, whitespace, other character behavior * Add help and fish mode to e2e interactive mode. (#3313) ## Summary Minor improvements to e2e.sh interactive mode: * add to -h output * do not run start stop test in interactive mode * support fish shell ## Test Plan Manual testing: ``` ~$ ./e2e.sh -i ... lots of output removed ... ********** READY ********** The test environment is now set. You can now run tests in another terminal. Configure the environment: set -g VIRTUAL_ENV "/home/will/go/src/github.com/algorand/go-algorand/tmp/out/e2e/130013-1639576513257/ve" set -g PATH "$VIRTUAL_ENV/bin:$PATH" python3 "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_client_runner.py "/home/will/go/src/github.com/algorand/go-algorand/test/scripts"/e2e_subs/SCRIPT_FILE_NAME Press enter to shut down the test environment... ``` * Minimum Account Balance in Algod (#3287) * Add convertAddress tool. (#3304) ## Summary New tool: convertAddress I share this tool with someone every few months, putting it in the repo along with some documentation should make it easier to share and encourage people to share it amongst themselves if it's useful. Merge `debug` into `tools` to make it easier to organize these miscellaneous tools. * tealdbg: increase intermediate reading/writing buffers (#3335) ## Summary Some large teal source files cause the tealdbg/cdt session to choke. Upping the buffer size to allow for larger source files. closes #3100 ## Test Plan Run tealdbg with a large source teal file, ensure the source file makes it to cdt without choking. * Adding method pseudo op to readme (#3338) * add config.DeadlockDetectionThreshold (#3339) Summary This allows for the deadlock detection threshold to be set by configuration. Test Plan Existing tests should pass. * Fix flaky test in randomized ABI encoding test (#3346) * update abi encoding test random testcase generator, scale down parameters to avoid flaky test * parameterized test script * add notes to explain why flaky test is eliminated * show more information from self-roundtrip testing * fully utilize require, remove fmt * ledger: perform the catchpoint writing outside the trackers lock. (#3311) ## Summary This PR moves the catchpoint file writing to be performed outside of the trackers lock. This resolves the issue where a long catchpoint file writing blocks the agreement from validating and propagating votes. ## Test Plan * [x] Test manually & use existing tests. * [x] Implement a unit test * [x] Deploy a local network where the catchpoint writing takes a long time and verify it doesn't get blocked during catchpoint writing. * Bump version number * Update `goal app method` handling of args and return values (#3352) * Fix method call arg overflow handling * Only check last log for return value * Address feedback * Add comment explaining ABI return prefix * Support app creation in `goal app method` (#3353) * Support app creation in `goal app method` * Don't use nonprintable tab character * Link to specific gist version * Fix error messages * Rename `methodCreation` to `methodCreatesApp` * Update license to 2022 (#3360) Update license on all source files to 2022. * Add totals checks into acct updates tests (#3367) ## Summary After #2922 there is some leftover unused code for totals calculations. Turned this code into actual asserts. ## Test Plan This is tests update * remove buildnumber.dat * PKI State Proof Incremental Key Loading (#3281) ## Summary Followup to #3261 (contained in diff). Use the new key loading routine from the REST API. ## Test Plan New unit tests. * Limit number of simultaneous REST connections (#3326) ## Summary This PR limits the number of simultaneous REST connections we process to prevent the exhaustion of resources and ultimately a crash. Two limits are introduced: soft and hard. When the soft limit is exceeded, new connections are returned the 429 Too Many Requests http code. When the hard limit is exceeded, new connections are accepted and immediately closed. Partially resolves algorand/go-algorand-internal#1814. ## Test Plan Added unit tests. * Use rejecting limit listener in WebsocketNetwork. (#3380) ## Summary Replace the standard limit listener with the new rejecting limit listener in `WebsocketNetwork`. This will let the dialing node know that connection is impossible faster. ## Test Plan Probably not necessary. * Delete unused constant. (#3379) ## Summary This PR deletes an unused constant. ## Test Plan None. * Add test to exercise lookup corner cases (#3376) ## Summary This test attempts to cover the case when an accountUpdates.lookupX method can't find the requested address, falls through looking at deltas and the LRU accounts cache, then hits the database — only to discover that the round stored in the database (committed in `accountUpdates.commitRound`) is out of sync with `accountUpdates.cachedDBRound` (updated a little bit later in `accountUpdates.postCommit`). In this case, the lookup method waits and tries again, iterating the `for { }` it is in. We did not have coverage for this code path before. ## Test Plan Adds new test. * Test for catchup stop on completion (#3306) Adding a test for the fix in #3299 ## Test Plan This is a test * Delete unused AtomicCommitWriteLock(). (#3383) ## Summary This PR deletes unused `AtomicCommitWriteLock()` and simplifies code. ## Test Plan None. Co-authored-by: DevOps Service <devops-service@algorand.com> Co-authored-by: Jason Paulos <jasonpaulos@users.noreply.github.com> Co-authored-by: chris erway <51567+cce@users.noreply.github.com> Co-authored-by: Shant Karakashian <55754073+algonautshant@users.noreply.github.com> Co-authored-by: John Lee <64482439+algojohnlee@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Ben Guidarelli <ben.guidarelli@gmail.com> Co-authored-by: Pavel Zbitskiy <65323360+algorandskiy@users.noreply.github.com> Co-authored-by: Jack <87339414+algojack@users.noreply.github.com> Co-authored-by: John Lee <john.lee@algorand.com> Co-authored-by: algobarb <78746954+algobarb@users.noreply.github.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> Co-authored-by: Fabrice Benhamouda <fabrice.benhamouda@normalesup.org> Co-authored-by: Tsachi Herman <tsachi.herman@algorand.com> Co-authored-by: Tolik Zinovyev <tolik@algorand.com> Co-authored-by: egieseke <eric_gieseke@yahoo.com> Co-authored-by: Quentin Kniep <kniepque@hu-berlin.de> Co-authored-by: Hang Su <87964331+ahangsu@users.noreply.github.com> Co-authored-by: Or Aharonee <or.aharonee@algorand.com> Co-authored-by: Barbara Poon <barbara.poon@algorand.com>
- Loading branch information
21 people
authored
Jan 7, 2022
1 parent
ef0fad7
commit 0dd28f2