Every push to master causes 2-3hr of downtime on ITRB CI

[edeutsch]

For quite some time, it has been noticed that every push to master causes 2-3 hr of downtime while arax.ci.transltr.io automatically refreshes.

One hypothesis is that the instance is needlessly downloading the databases every time.

[edeutsch]

After getting help from Pouyan on the issue, he confirmed that it is redownloading the KG2.9.0 databases with every triggered redeployment.

The root cause was determined to be that the MD5 checksums are not where they are supposed to be. The current state is:

Remote working directory: /transltr-063208468694-sftp/team-expander/md5_sums
-rwxr--r--   1        -        -       85 Apr 15 15:47 ExplainableDTD_v1.0_KG2.8.4_refreshedTo_KG2.9.0.db.md5
drwxr--r--   1        -        -        0 Aug  4  2022 KG2.7.6
drwxr--r--   1        -        -        0 Jan 12  2023 KG2.8.0
drwxr--r--   1        -        -        0 May 24  2023 KG2.8.0.1
drwxr--r--   1        -        -        0 Aug 17  2023 KG2.8.4
drwxr--r--   1        -        -        0 Apr 15 15:46 KG2.9.0
-rwxr--r--   1        -        -       67 Apr 15 15:47 autocomplete_v1.2_KG2.9.0.sqlite.md5
-rwxr--r--   1        -        -       69 Apr 15 15:47 curie_to_pmids_v1.0_KG2.9.0.sqlite.md5
-rwxr--r--   1        -        -       74 Apr 15 15:47 fda_approved_drugs_v1.2_KG2.9.0c.pickle.md5
-rwxr--r--   1        -        -       59 Apr 15 15:47 kg2c_v1.2_KG2.9.0.sqlite.md5
-rwxr--r--   1        -        -       61 Apr 15 15:47 meta_kg_v1.2_KG2.9.0c.json.md5
-rwxr--r--   1        -        -       71 Apr 15 15:47 node_synonymizer_v1.2_KG2.9.0.sqlite.md5

the KG2.9.0 MD5sums are supposed to be in the KG2.9.0 folder, but instead they are in the parent directory.
When the download script cannot find the MD5sums, it just redownload all those databases every single time.

I will attempt to move the MD5sums to the correct location.

[saramsey]

This has been fixed, right? (and thank you for that, @edeutsch !) Can we close it out?

[edeutsch]

Yes, ITRB fixed this. Closing.