An error in kp_selector.py

[chunyuma]

Hi team,

I ran pytest -vk test_ARAX_infer.py to test ARAX_infer with the updated xDTD database. Most of tests have passed except for test_xdtd_with_only_qg. The error message seems to indicate the error is from kp_selector.py. I pasted the error mssage below:

2024-03-17T17:37:08.315013 INFO: (1634983) [] ARAX Query launching on incoming Query
2024-03-17T17:37:08.315063 DEBUG: (1634983) [] RTXConfiguration says maturity=development, current_branch=master, is_itrb_instance=False, arax_version=1.4.3, trapi_version=1.4.0
2024-03-17T17:37:08.315091 INFO: (1634983) [] Creating an empty template TRAPI Response
2024-03-17T17:37:08.315159: DEBUG: In BiolinkHelper init
2024-03-17T17:37:08.318139: DEBUG: Loading BL lookup map...
2024-03-17T17:37:08.318179: DEBUG: Loading pickle file: /scratch2/cqm5886/RTX/code/ARAX/BiolinkHelper/biolink_lookup_map_3.1.2_v4.pickle
2024-03-17T17:37:08.320308: DEBUG: Done loading BL lookup map
2024-03-17T17:37:08.323424 INFO: (1634983) [] Examine input Query for needed information for dispatch
2024-03-17T17:37:08.323460 INFO: (1634983) [] Validating the input query graph
2024-03-17T17:37:08.323489 DEBUG: (1634983) [] Deserializing message
2024-03-17T17:37:08.323685 INFO: (1634983) [] Found input query_graph. Interpreting it and generating ARAXi processing plan to answer it
2024-03-17T17:37:08.396211 DEBUG: (1634983) [] Assessing the QueryGraph for basic information
2024-03-17T17:37:08.396251 DEBUG: (1634983) [] Found 2 nodes and 1 edges
2024-03-17T17:37:08.396975 DEBUG: (1634983) [] canonical_curies={'MONDO:0015564': {'preferred_curie': 'MONDO:0015564', 'preferred_name': 'Castleman disease', 'preferred_category': 'biolink:Disease', 'all_categories': {'biolink:Disease': 13}}}, curie=MONDO:0015564
2024-03-17T17:37:08.397057 DEBUG: (1634983) [] The QueryGraph reference template is: n00(ids)-e00()-n01(categories=biolink:ChemicalEntity)
2024-03-17T17:37:08.397836 INFO: (1634983) [] Found input processing plan. Sending to the ProcessingPlanExecutor
2024-03-17T17:37:08.397868 DEBUG: (1634983) [] Entering execute_processing_plan
2024-03-17T17:37:08.397941 DEBUG: (1634983) [] Connecting to ResponseCache
2024-03-17T17:37:08.399836 DEBUG: (1634983) [] Creating ARAXMessenger instance
2024-03-17T17:37:08.399874 DEBUG: (1634983) [] A single Message is ready and in hand
2024-03-17T17:37:08.400304 DEBUG: (1634983) [] Saving original query graph (has qnodes {'chemical', 'disease'} and qedges {'t_edge'})..
2024-03-17T17:37:08.400331 DEBUG: (1634983) [] Found actions
2024-03-17T17:37:08.400364 INFO: (1634983) [] Parsing input actions list
2024-03-17T17:37:08.400387 DEBUG: (1634983) [] Parsing action: expand()
2024-03-17T17:37:08.400441 DEBUG: (1634983) [] Parsing action: overlay(action=compute_ngd, virtual_relation_label=N1, subject_qnode_key=disease, object_qnode_key=chemical)
2024-03-17T17:37:08.400497 DEBUG: (1634983) [] Parsing action: filter_kg(action=remove_general_concept_nodes,perform_action=True)
2024-03-17T17:37:08.400542 DEBUG: (1634983) [] Parsing action: resultify()
2024-03-17T17:37:08.400581 DEBUG: (1634983) [] Parsing action: filter_results(action=limit_number_of_results, max_results=500)
2024-03-17T17:37:08.400673: DEBUG: In BiolinkHelper init
2024-03-17T17:37:08.403481: DEBUG: Loading BL lookup map...
2024-03-17T17:37:08.403521: DEBUG: Loading pickle file: /scratch2/cqm5886/RTX/code/ARAX/BiolinkHelper/biolink_lookup_map_3.1.2_v4.pickle
2024-03-17T17:37:08.405541: DEBUG: Done loading BL lookup map
2024-03-17T17:37:08.405781 INFO: (1634983) [] Processing action 'expand' with parameters {'': 'true'}
2024-03-17T17:37:08.405995 ERROR: (1634983) [UnknownError] Unable to load KP info caches: KP info cache(s) do not exist.
2024-03-17T17:37:08.406024 DEBUG: (1634983) [] Loading cached Smart API amd meta map info
2024-03-17T17:37:08.406065 ERROR: (1634983) [LoadKPCachesFailed] Failed to load KP info caches due to [Errno 2] No such file or directory: '/scratch2/cqm5886/RTX/code/ARAX/ARAXQuery/Expand/cache_smart_api_and_meta_map_1.4--development.pkl'
2024-03-17T17:37:08.407780 ERROR: (1634983) [UncaughtARAXiError] An uncaught error occurred: 'NoneType' object has no attribute 'keys': ['Traceback (most recent call last):\n', '  File "/scratch2/cqm5886/RTX/code/ARAX/test/../ARAXQuery/ARAX_query.py", line 766, in execute_processing_plan\n    expander.apply(response, action[\'parameters\'], mode=mode)\n', '  File "/scratch2/cqm5886/RTX/code/ARAX/test/../ARAXQuery/ARAX_expander.py", line 152, in apply\n    kp_selector = KPSelector(kg2_mode=True, log=log) if mode == "RTXKG2" else KPSelector(log=log)\n', '  File "/scratch2/cqm5886/RTX/code/ARAX/ARAXQuery/Expand/kp_selector.py", line 29, in __init__\n    self.valid_kps = {"infores:rtx-kg2"} if self.kg2_mode else set(self.kp_urls.keys())\n', "AttributeError: 'NoneType' object has no attribute 'keys'\n"]

Can anyone help with this issue? Thanks!

[amykglen]

hey @chunyuma - I think ever since the changes made last summer to how Expand refreshes its KP info cache, if you're running ARAX locally, you need to manually run the KP info cache process before running tests (I think the cache needs to be refreshed within the last 24 hours or you'll see that error). you can do that by navigating to the /RTX/code/ARAX/ARAXQuery/Expand directory and running:

python kp_info_cacher.py

then you should be able to run pytests as usual.

it'd be great if we somehow came up with a way of automatically running this command prior to running pytests locally (the process is automatically run by the BackgroundTasker on instances that are running the whole ARAX flask application, but when we run tests locally we're not actually running flask, so there's no BackgroundTasker..)

[dkoslicki]

Before a solution that runs automatically is implemented, do you think it would be worth adding such directions to the wiki in the Dev notes about local running of ARAX?

[chunyuma]

Thanks @amykglen! It fixed the issue.

I will keep this issue open to discuss if we need to add the directions to the wiki or make an automatic KP info cache.

[amykglen]

Before a solution that runs automatically is implemented, do you think it would be worth adding such directions to the wiki in the Dev notes about local running of ARAX?

Yes, I do. @sundareswarpullela - can you update the wiki with this step since you made the changes to the KP info caching process?

[sundareswarpullela]

@amykglen I will add this ASAP

[sundareswarpullela]

Its already mentioned in step 5 of Setting up for local dev work on ARAX

[amykglen]

@sundareswarpullela - ah, nice, thanks! maybe it would make sense to add a note to the pytest section too, since it needs to be run basically any time we run pytests? (not just the initial dev environment set up)

https://github.com/RTXteam/RTX/wiki/Dev-info#arax-pytest-suite

[sundareswarpullela]

Updated the dev info. Closing the issue.