Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rare KeyError while reading aggregates #3384

Open
jessebrennan opened this issue Aug 30, 2021 · 3 comments
Open

Rare KeyError while reading aggregates #3384

jessebrennan opened this issue Aug 30, 2021 · 3 comments
Labels
bug [type] A defect preventing use of the system as specified debt [type] A defect incurring continued engineering cost indexer [subject] The indexer part of Azul orange [process] Done by the Azul team spike:1 [process] Spike estimate of one point workaround [type] An enhancement that works around a defect of an external dependency

Comments

@jessebrennan
Copy link
Contributor 

[WARNING] 2021-08-27T17:00:03.486Z 84490e63-5d43-5a06-b1bb-57c04110efe6 Failed to aggregate tallies: dict_values([[DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='c55da389-1589-49f2-b748-8974bf39fcb6', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='f0a547f1-c4a6-4834-bc7b-ad8ef49b8f90', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='cell_suspensions', entity_id='2b12c961-33f0-496d-a4c2-ecacbcdc1087', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='samples', entity_id='00a5be43-cc47-49d1-ae78-8ded0a3ad18f', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='bundles', entity_id='1b163269-4e17-4376-89b9-e2b0ea9954b3', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='58d289bd-a4d0-4ed6-bfd8-1c992da7ce1e', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='e97b5980-8d41-4bd2-999a-2fd81ac79a20', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='bundles', entity_id='8886f59c-dbe9-4431-8aa3-2e39d95d1d5a', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='0610eb4a-1f87-49cb-90ad-ccc57c43a8f4', catalog='dcp8'), num_contributions=1, attempts=1)], [DocumentTally(entity=CataloguedEntityReference(entity_type='files', entity_id='53738b66-334d-4250-8ab7-030b26035c18', catalog='dcp8'), num_contributions=1, attempts=1)]])

Traceback (most recent call last):
File "/var/task/azul/indexer/index_controller.py", line 229, in aggregate
self.index_service.aggregate(tallies)
File "/var/task/azul/indexer/index_service.py", line 236, in aggregate
old_aggregates = self._read_aggregates(tallies)
File "/var/task/azul/indexer/index_service.py", line 307, in _read_aggregates
return {a.coordinates.entity: a for a in aggregates()}
File "/var/task/azul/indexer/index_service.py", line 307, in <dictcomp>
return {a.coordinates.entity: a for a in aggregates()}
File "/var/task/azul/indexer/index_service.py", line 299, in aggregates
if doc['found']:
KeyError: 'found'
@jessebrennan jessebrennan added the orange [process] Done by the Azul team label Aug 30, 2021
@melainalegaspi melainalegaspi added bug [type] A defect preventing use of the system as specified workaround [type] An enhancement that works around a defect of an external dependency code [subject] Production code and removed bug [type] A defect preventing use of the system as specified labels Aug 31, 2021
@melainalegaspi
Copy link

Spike to confer with ES team about whether this is expected.

@melainalegaspi melainalegaspi added the spike:1 [process] Spike estimate of one point label Aug 31, 2021
@jessebrennan
Copy link
Contributor Author

https://discuss.elastic.co/t/rarely-the-found-key-is-missing-from-the-multi-get-response/283584?u=jesseb

@jessebrennan
Copy link
Contributor Author

Once we fix #3312, we will be able to search through the logs and find the response.

@melainalegaspi melainalegaspi added the debt [type] A defect incurring continued engineering cost label Sep 8, 2021
@melainalegaspi melainalegaspi added the bug [type] A defect preventing use of the system as specified label Sep 30, 2021
@theathorn theathorn added indexer [subject] The indexer part of Azul and removed code [subject] Production code labels Nov 5, 2021
hannes-ucsc added a commit that referenced this issue Nov 24, 2021
@hannes-ucsc
Better handle ES mget response items lacking the found key (#3384)
c5adabd
achave11-ucsc added a commit that referenced this issue Nov 30, 2021
@achave11-ucsc
Backport: 45ffffb (#3659, #3384, PR #3666, PR #3674)
78d6587
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug [type] A defect preventing use of the system as specified debt [type] A defect incurring continued engineering cost indexer [subject] The indexer part of Azul orange [process] Done by the Azul team spike:1 [process] Spike estimate of one point workaround [type] An enhancement that works around a defect of an external dependency
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jessebrennan @theathorn @melainalegaspi