Transaction closed before all data is fetched when working in FIRESTORE_AS_DATASTORE mode

[dmitrykuzmin]

When reading from the message storage transactionally and in bulk, the transaction wraps only the the process of obtaining a DsQueryIterator. The actual iteration process occurs later at the will of the user.

Most of the time this works fine, as the DsQueryIterator performs a call to the Datastore immediately on its initialization, obtaining QueryResults which supposedly already hold the queried dataset.

The problem is, as QueryResults docs state, the data is actually loaded lazily in batches, and the size of these batches is determined by the Datastore itself. The batch size doesn't have to match the limit passed with the query params and cannot be controlled by the user.

In case of discrepancy, one or more additional reads from the Datastore will be attempted when iterating over the DsQueryIterator. As the transaction is already closed, the reads will fail with the The referenced transaction has expired or is no longer valid. message.