Improve robustness of Spacewalk integration tests

[ebma]

Closes #494.

The multiple mainnet integration issues (and its fixes) are as follows:

• cannot determine which DecimalsLookUp to use.
  • fix: make sure for the feature integration-test, the clients/vault uses the standalone-metadata feature:
    

    spacewalk/clients/vault/src/lib.rs

    Lines 52 to 53 in ea913fa

    	if #[cfg(feature = "standalone-metadata")] {
    	pub type DecimalsLookupImpl = primitives::DefaultDecimalsLookup;

  • fix: make sure the clients/runtime has access to the mainnet-runtime
• failed with

     HorizonResponseError(reqwest::Error { kind: Decode, source: Error("invalid type: string \"ySHX => SHX/XLM DAILY\", expected a borrowed string", line: 9484, column: 44) })
  

  • fix: Some memo texts have non-literal characters, messing up deserialization. A solution is to use Cow<str> instead String.
• 'could not find event: Issue::ExecuteIssue' potentially because all transactions have been traversed (reached EOF) before the memo to check hashmap was filled. A.K.A. probably race condition.
  • fix: DO NOT ITERATE OVER the list of transactions UNLESS the hashmap has values inside.
    • introducing IsEmptyExt since the parameters being passed are using generic datatypes.

           pub trait IsEmptyExt {
           fn is_empty(&self) -> bool;
           }

      spacewalk/clients/wallet/src/horizon/horizon.rs

      Lines 255 to 257 in ea913fa

      	issue_map: Arc<RwLock<T>>,
      	memos_to_issue_ids: Arc<RwLock<U>>,
      	filter: impl FilterWith<T, U>,

[b-yap]

one test is failing with:

thread 'test_redeem_succeeds_on_mainnet' panicked at 'should return ok: HorizonSubmissionError { title: "Transaction Failed", status: 400, reason: "tx_failed", result_code_op: ["op_underfunded"], envelope_xdr: Some("AAAAAgAAAACne8I5bB4KhDOubC5DCq7O/p4yvnXbjNkOE8fT3v0MHwAAAGUC7t8CAAABLwAAAAAAAAABAAAAHDNCMWdEd2U1YTZUS2dFcHVmdmJZcmRCcVZWNHYAAAABAAAAAQAAAACne8I5bB4KhDOubC5DCq7O/p4yvnXbjNkOE8fT3v0MHwAAAAEAAAAArwuPZBi/hC2gPFK8R7thoU5Xsbh+k5n1ls8DrCkdhJMAAAABVVNEQwAAAAA7mRE4Dv6Yi6CokA6xz+RPNm99vpRr7QdyQPf2JN8VxQAAAAAAAYagAAAAAAAAAAHe/QwfAAAAQElB9VXUO8NynqXVQVrdXuAECle2Iebo48ciDX+d5T0sMq3kz7YVDQqcw3uvPftSmCtRXLDoP2sAleJBIcNopQg=") }', clients/vault/tests/helper/helper.rs:196:6

the address GCTXXQRZNQPAVBBTVZWC4QYKV3HP5HRSXZ25XDGZBYJ4PU667UGB642R doesn't have any balance anymore.

[ebma]

Let's hope that this finally fixes it.

[gianfra-t]

This sleep is ultimately for poll_messages_from_stellar process to have time for writing enough messages into ScpMessageCollector?

[b-yap]

Yes, to build the proof.

[ebma]

Now the CI job failed with

thread 'test_redeem_succeeds_on_mainnet' panicked at 'should return ok: HorizonSubmissionError { title: "Transaction Failed", status: 400, reason: "tx_insufficient_fee", result_code_op: [], 

which is weird because recently already changed the default fee percentile to be 95 instead of 90. We could set it to 100 but then we potentially pay a very large fee :(

[b-yap]

@ebma
I think it might do us good if we have more than 2 secret keys for testing; Like 5 keys (and I meant on testnet network).
Separate keys per lib/module/package.
There's the stellar-relay-lib, wallet, vault.

[ebma]

It might help yes. But does it make sense? Shouldn't in a perfect world our tests be able to just work with one set of accounts instead of us having to rely on switching them randomly?

[b-yap]

you are right. :/

[ebma]

It looks good to me. Do you think you found the problem with macOS or was it more or less a coincidence that it passed this time?

[ebma]

Did you notice a meaningful difference with this change?

[b-yap]

To be completely honest, I am unsure.

However we encountered idle connections before and even talked about using ClientBuilder. I guess now's the right time.

I set the pool to half of default because zero is too huge a leap.

[b-yap]

@ebma For macos it's definitely a coincidence.
I'll be keeping my eye on the mainnet tests.

[ebma]

I know I already brought this up a very long time ago, but how about we just remove the macOS job from the workflow completely? I don't see much of a benefit here, besides our workflows taking twice the time and being more likely to fail in some sense. Of course, removing something just because it doesn't work is not the ideal solution but generally, even assuming that the macOS job worked reliably all the time, I'd argue that vault operators would deploy these vaults in a Linux environment and not on a macOS machine so it should be fine to only test the workflow on Ubuntu.

WDYT @b-yap? Do you want to keep it or should we remove it?

[b-yap]

@ebma
I'd vote on removing it.

[ebma]

Let's remove it and merge this without waiting for the checks