loosing some index updates to the cluster

[oborder]

Hi

when I do testing of index insertion with local es instance all models are properly indexed in the es,
but when using cluster if the model insertion rate is too high some updates get lost.
I don't know much about akka and rabbitmq but i suspect they get lost somewhere in there and are
never posted to the es cluster.

You can test it easily by creating a few thousand models in a loop, I can provide test case if needed.

[oborder]

also is it possible to handle play shutdown so the jobs in the queue in ElasticSearchIndexer.stream will be preserved?
Otherwise ti may be pretty hard to ensure all saved db entites made it to es index...

[bgooren]

Hmmm, I haven't used the code path which uses a message bus, but my first hunch is that the message bus is dropping messages when the channel overflows. In this case the producer (this module) is faster than the consumer (es), and I've seen settings on channels to drop messages when overflow occurs. Can you please check those settings?

WRT preserving the local job queue: I think we can look into that. Shouldn't be too hard to serialize them to disk.