Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up Tests #877

Closed
jsdw opened this issue Mar 21, 2023 · 2 comments · Fixed by #944
Closed

Speed up Tests #877

jsdw opened this issue Mar 21, 2023 · 2 comments · Fixed by #944
Assignees

Comments

@jsdw
Copy link
Collaborator

jsdw commented Mar 21, 2023

Tests (particularly the integration tests I think) take >20mins to complete; can we speed this up? Does cargo nextest help at all (did we try it and find it didn't?). Any other way to parallelise tests? Worth running them on Gitlab or something instead? It'd be amazing if we could get this to under 5 mins :)

@jsdw
Copy link
Collaborator Author

jsdw commented Apr 25, 2023

#928 really helped and has sped things up a fair bit.

Other things we can investigate:

  • Fewer UI tests; we generate a UI test to generate every pallet individually. Do we need them all? Might be better to test specific pallets instead. This should be a fairly easy win at the mo, since UI tests can take longest.
  • We use the #[subxt] macro a lot. How long does it take per invocation? Can we speed up most of our tests/examples by pointed at a stripped down version? Maybe a artifacts/polkadot_metadata_tiny.scale one with no pallets in that can be used anywhere we don't need any pallet logic. Maybe also a artifacts/polkadot_metadata_small.scale for examples that only need a few pallets in. Maybe rename the other to polkadot_metadata_full.scale for consistency, if it's still actually used somewhere. (add readme to artifacts folder to describe exactly what is in each file).
  • Try cargo nextest again; with faster machines, maybe we can get more speedup running tests with this? Maybe we can merge test runs and rely more on this (plus possibly a more powerful machine) to just run everything in one place in parallel?

Probably more painful and less wins:

  • Is there a way to cache the output from the #[subxt] macro when inputs are the same?
  • Is there any win from sharing a Substrate instance in some of the integration tests rather than spinning up a new one for every test? Probably a fair difficult thing to try; need to ensure tests use unique accounts for instance (not just alice any bob) and only applicable to tests that can be run alongside others without any conflict.

@tadeohepperle
Copy link
Contributor

@jsdw I found out that invocation of the subxt macro takes about 110ms on my machine for the polkadot.scale metadata that is 346KB in size. I created a stripped down version of the default substrate metadata (without any pallets) that is just 6.43KB in size. It reduces the time of macro invocation to about 2.5ms. This is a pretty linear speed up (50x smaller -> 50x faster). Hoverever this is just the macro invocation which is, (just my guess) probably a lot times faster than the actual compilation of the generated code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants