Retries are dangerous for scrolled search queries

[jordanlibrande]

Retrying a scrolled search query that started executing, but failed on timeout/connection issue will lead to that page being skipped, thus leading to incomplete results across the entire set of queries.

I'm not sure exactly what the library improvement would be, but I'm hoping to improve the library somehow so other people don't get hit with this unexpected behavior.

In my use case, I would always prefer to error out, rather than skip a page of results. I'm planning to work around this on my end by creating a custom retrier for scrolled search, this retrier will retry on "node not available" and some similar errors, but will not retry on timeout errors. Then I'll create two elastic clients, one for normal search with a normal retrier, and one for scrolled search with the custom scrolled retrier.

Eventually this core issue of scrolled search being bad might be fixed in a later ES version via something like elastic/elasticsearch#26472

Possible approaches here could include:

• better package documentation for retriers about the danger of retrying scrolled search
• another new built-in retrier that only retries on "safe" errors
• client stores separate retrier objects for use in normal and scrolled searches

[olivere]

Thanks for bringing this up. I will tinker with this as well. Maybe we find some good solutions that everyone is happy with.

[olivere]

I think what could work is to

a) use a special built-in retrier for ScrollService that only retries on "safe" errors (skipping the default retrier on the Client),
b) allow users to configure a custom retrier in ScrollService, replacing the default one in a), and
c) make Client.PerformRequest accept a Retrier (using the retrier on the Client by default).

ScrollService will pass its custom/configured Retrier to Client.PerformRequest, while other services will use the default.

[olivere]

I just released 5.0.60 and 6.1.1 which both allow to set a per-request Retrier (with configurable Backoff). For now, only Bulk API and Scroll API make use of that. Use Retrier(...) on the BulkService or ScrollService to pass your own retrier for the given service.